Modal disclosed their Coding Agent inference service optimization practice, based on Kimi K2.6, SGLang and B200, which improves single-user Decode performance by approximately 2.8 times and Replica (single model service instance) throughput by approximately 5.6 times.
This time, the inference performance optimization mainly focuses on three parts: Decode performance, KV Cache under long sessions, and routing and load balancing in multi-session scenarios. This article will develop along these three parts and add some engineering practices in the same direction.
Legend: Schematic diagram of context accumulation in Coding Agent Session round by round. Picture source @Modal
Decode performance optimization
For Coding Agent, Decode TPS will directly affect the advancement speed of the task. Tool calling parameters, code modifications, and intermediate analysis all require models to be generated token by token. The slower the Decode, the later the subsequent tool execution and the next round of inference will start.
Under the original configuration, once the concurrency exceeds 6 users, the Decode TPS and overall throughput of a single user will decrease significantly. Even at low concurrency, the build speed is not as fast as Modal expected.
Therefore, when it comes to Decode optimization, they first focus on the Decode TPS actually obtained by a single user.
In the Decode stage, every time the model generates a Token, a large number of model weights and states must be read. For this scenario, HBM bandwidth often becomes a bottleneck before computing power. There are two ways to optimize here: Tensor Parallelism (abbreviated as TP below) and Speculative Decoding.The former splits a model calculation into multiple sheets GPU On, the latter attempts to confirm multiple Tokens in one forward calculation of the target model。
TP4 or TP8
The model weight of Kimi K2.6 occupies approximately 595 GB, while a single B200 provides 180 GB HBM. Therefore, neither TP1 nor TP2 can run the complete model without unloading the weights, and the optional configurations mainly fall on TP4 and TP8.
Legend: Comparison of HBM and KV Cache capacity of TP4/TP8. Picture source @Modal
TP4 uses 4 B200s, the total HBM is 720 GB, and about 125 GB is left after deducting the model weight; TP8 uses 8 B200s, the total HBM can reach 1.44 TB, and the remaining space is about 845 GB. According to the estimated BF16 KV configuration at the time, TP4 can accommodate approximately 450,000 Tokens in KV, while TP8 can accommodate approximately 3 million Tokens.
Looking at the video memory capacity alone, the advantages of TP8 are very obvious. However, under the actual load of Modal, even if TP8 is only responsible for Prefill, the throughput of each GPU is only roughly equal to TP4, which handles both Prefill and Decode. Doubling the number of GPUs does not bring corresponding unit GPU benefits.
In the end Modal chose TP4 and used the smaller Replica as the expansion unit. Although this choice improves the efficiency of unit GPU usage, it also compresses the space left for KV Cache: after deducting the model weight, TP4 only has about 125 GB HBM left. When concurrency continues to be increased in the future, KV Cache will soon become a new bottleneck.
Load adaptation of Speculative Decoding
After the TP configuration is determined, you can continue to optimize Speculative Decoding.
Modal uses DFlash as the speculative model. The speculative model will first predict the next multiple Tokens, and then be uniformly verified by the target model. As long as it can accept multiple prediction results at one time, the target model can generate multiple valid tokens after reading a large number of weights and KVs in each round, thereby reducing the inference overhead of a single token.
The output of Coding Agent is more suitable for this type of optimization. There are a lot of repeated structures in the code, tool calls usually use fixed formats, and chat templates also contain many control tokens; the Agent will also frequently reuse paths, function names, and code snippets that have just appeared. These contents are more predictable and more likely to increase the acceptance rate of Speculative Decoding.
Modal also does not directly follow the general DFlash speculation model. They first trained with general data, and then continued fine-tuning using the Coding Trace actually generated by the target model to make the inferred model closer to the output distribution of the actual Coding Agent.
After fine-tuning, the average acceptance length on the representative Trace ranges from 5.00 increased to 5.84 Token/stepand additionally brought approx. 20% speed increase. In order to continue to use online data to train the inference model, Modal also modified SGLang to allow the service to directly record the original Token ID, preventing the original Token sequence from being completely restored after the text is re-segmented.
Compared with simply accessing Speculative Decoding, what is more valuable in this part is the adaptation of the output of the real Coding Agent. How much benefit a speculative model can bring depends largely on whether the training data is consistent with the actual request distribution.
Together AI’s practice in production reasoning follows a similar line of thinking. They use specially trained inferential models and incorporate adaptive acceptance mechanisms. Their public data shows thatMore predictable workloads, such as code and structured output, are often more likely to benefit from Speculative Decoding.. Aurora, launched in 2026, will further utilize online inference trajectories to continuously adjust the speculation model to adapt to changes in production traffic.
Therefore, in the Coding Agent scenario, the effect of Speculative Decoding cannot only be based on the benchmark results of the fixed speculative model. Different proportions of tool calls, code, and natural language in Agent output will affect the final average acceptance length and actual acceleration effect.
KV Cache expansion under long session
After TP4 and DFlash improved single-user Decode performance, the new bottleneck turned to HBM capacity: there was not enough space left for Session status and KV Cache.
Under actual load with a maximum input length of about 100k Tokens, the KV space of TP4 can only accommodate approximately 4 active sessions. After continuing to increase concurrency, the KV Cache hit rate will drop rapidly, and the long prefixes that have been calculated need to be re-prefilled, and the throughput and interaction speed will also drop accordingly.
The next optimization focus basically revolves around one question:Why in limited HBM leaving more space for KV Cache。
DFlash’s video memory usage optimization
The next thing to deal with first is the memory usage of DFlash in SGLang.
The original implementation will first retain the pointer to the intermediate results of the target model, and then copy these data to continuous memory after a forward calculation is completed. During this process, the original data and the buffer used to store the results will occupy the video memory at the same time.
The optimized process is changed to allocate continuous buffers in advance and directly write intermediate results during the forward calculation process, ultimately reducing the peak HBM usage in this part by half. Relevant modifications were later submitted to SGLang.
This type of change is not as noticeable as quantization or new operators, but the released HBM can be directly left in the KV Cache to make room for subsequent improvements in concurrency.
KV Cache quantization: BF16 KV replaced by FP8
Compared with the previous memory optimization, the KV Cache capacity improvement brought by quantization is more obvious.
After passing the actual evaluation, KV Cache was changed from BF16 to FP8. The storage overhead of a single KV element is halved, and the same HBM can hold approximately twice the Token.
At the same time, the shared experts originally saved in FP8 and activated for each Token have also been changed to NVFP4, continuing to release part of HBM. In relevant evaluations, no obvious quality degradation beyond the operational fluctuation range caused by these modifications was observed.
For long sessions, the unit storage overhead of KV Cache will directly affect how much historical state a Replica can retain, and will further affect cache hits and concurrency capabilities.
This direction continues to develop toward lower precision. NVIDIA subsequently tested NVFP4 KV Cache on Qwen3-Coder-480B-A35B. Compared with FP8, KV Cache occupancy can be further reduced by half; in its test environment, the cache hit rate is increased by approximately 20%the highest latency improvement is approximately 3 times. As cache capacity becomes sufficient, the gap between the two accuracies shrinks, and the final gain still depends on the model, context length, and actual load. NVIDIA reports less than 1% accuracy loss in tests such as LiveCodeBench, MBPP, Ruler 64K, and more.
The benefits brought by KV quantification are not only reflected in the memory usage, but also continue to affect the cache hit rate, concurrency and TTFT.
HiCache tiered cache
Even after memory cleaning and quantification are completed, HBM capacity is still limited. Next, SGLang’s HiCache was enabled, and the KV Cache was expanded from a single-layer HBM to a hierarchical cache: GPU HBM as the first layer, CPU memory as the second layer, and HiCache also supports the continued expansion of the third layer to distributed storage.
In this practice, the CPU memory is mainly used as the second layer cache to handle KV data that exceeds the HBM capacity in a short period of time. CPU memory capacity is larger, but the access speed is lower than HBM. If a large number of requests frequently read KV from CPU memory, the delay will also increase.
Therefore, this layer of cache is mainly used to buffer transient high concurrency: when a Replica exceeds the ideal concurrency for a short period of time, try to avoid a rapid decline in the KV Cache hit rate, resulting in a large number of long prefixes being re-prefilled.
Legend: KV Cache hit rate and performance changes as concurrency increases before and after HiCache is added. Picture source @Modal
Similar ideas can be extended to larger clusters. Together AI uses three layers of KV Cache in Cache-aware Prefill–Decode Disaggregation (CPD): GPU memory, host memory, and cluster-level distributed cache accessed through RDMA.
At the same time, requests will be divided into low-reuse and high-reuse categories according to the degree of KV reuse, allowing long requests that require complete prefill to be processed separately from requests that can be reused in the cache. In its testing, this solution improved the sustainable QPS by approximately 10% while meeting the tail latency requirements. 35%～40%。
HiCache mainly alleviates the problem of a single Replica exceeding the HBM capacity in a short period of time, while Together AI’s CPD continues to extend the reuse scope of KV Cache to the cluster level. For long context requests, whether the calculated KV can continue to be reused will directly affect subsequent Prefill overhead and overall throughput.
Multi-session routing and load balancing
With KV Cache, request routing cannot just look at which Replica is currently idle.
For a Coding Agent Session that has been running for dozens of rounds, if the next round of requests continues to enter the original Replica, a large number of previous contexts can directly reuse the local KV; if it is assigned to a Replica that has never processed this Session, the prefixes of hundreds of thousands of Tokens may need to be re-Prefilled.
Session Affinity was initially used: the client attaches the Modal-Session-Idthe router tries to fix subsequent requests for the same Session to the same Replica based on the Session ID.
This can improve the cache reuse rate, but it will also bring new scheduling problems under real Agent traffic.
Load skew for Session Affinity
The first problem comes from the hash allocation itself.
Assume that each Replica carries an average of 5 Sessions. Even if the overall number of Sessions is large enough, some Replicas may still be idle and some Replicas may be busy after random allocation. Take 250 Sessions allocated to 50 Replicas as an example. Although the average value is exactly 5, the actual load will still deviate significantly. In addition, the calculation amount of different Sessions is not completely consistent, and it is difficult to ensure that each Replica obtains a similar workload by only mapping based on Session ID.
Another problem comes from concurrency within the same Session.
Traditional conversations are mostly one question and one answer: after one request is completed, it goes to the next round. However, the Coding Agent may start Subagent, or may fork multiple concurrent tasks from the same context. At this time, multiple requests share the same Session ID. If they are still fixed to the same Replica through Session Affinity, it is easy to form a hotspot.
In response to this situation, concurrent requests for the same Session are no longer forced to fall on the same Replica, and some requests can be allocated to other instances. This will lose some KV Cache reuse, but it can lead to a more even load.
Therefore, in a long Session scenario, routing needs to make a choice between cache reuse and load balancing: when to continue to maintain Session Affinity, and when to allow requests to be distributed to other Replicas.
Cache reuse during capacity expansion
Automatic expansion will also bring about another type of cache reuse problem.
Traditional consistent Hash or Rendezvous Hash will remap part of the existing Session to the new Replica after adding a node. For stateless services, this is usually just a normal traffic rebalancing; but for Agent Sessions that have accumulated a large amount of KV, redistribution means that the cache on the original Replica cannot continue to be reused, and the new Replica needs to be re-Prefilled.
Therefore, when expanding capacity, priority will be given toNew Session Enter new Replicatry to keep the old Session that has established KV status in the original Replica. At the same time, routing will also determine the placement point of new requests based on the current load of Replica, running requests and other statuses.
After adjustment, the load distribution among Replica is significantly more even, the tail fluctuation of TTFT has also been reduced, and the actual throughput of multi-Replica deployment is closer to the level derived from the single-Replica benchmark test.
Legend: Replica request distribution before and after optimization. Picture source @Modal
Similar practices also appear in other Agent inference services. Cloudflare used on Workers AI x-session-affinitylet the same Session try to return to the location where its input state has been saved. Agent Harness such as OpenCode also added this request header.
According to the data disclosed by Cloudflare, after enabling Session Affinity in high-traffic scenarios, the cache hit rate of input tokens during peak periods has increased from approximately 60% increased to 80%. At the same time, its internal routing will also refer to the KV Cache status, because only knowing the Session ID is not enough to complete the final scheduling.
Session Affinity can improve the cache reuse rate, but after entering a multi-Replica scenario, routing also needs to consider the current location of the Session, the reuse value of the KV Cache, the real-time load of the Replica, and whether the same Session generates concurrent requests.
summary
Judging from these practices, it is difficult to completely separate Decode, KV Cache and multi-session scheduling. TP configuration will affect the video memory margin, KV Cache capacity will affect concurrency and routing, and Session Affinity will in turn affect cache reuse and load distribution.
Therefore, under real Coding Agent load, many optimizations need to be put back into the specific model, session size and traffic distribution for re-verification. The fixed configuration can be used as a reference, but the final effect must be confirmed by actual testing.
