Benchmarks: vLLM and SGLang recompute over half of agent history between tool calls

A benchmark study published on dev.to argues that vLLM and SGLang, two of the most widely used open-source inference engines, handle AI agent traffic poorly: under realistic load, more than half of the prompt tokens they processed were conversation history the engines had already computed once and then evicted from GPU memory.

The agent turn problem

According to the author, who publishes as nahuel990, a coding agent works in a loop: it sends the entire conversation to the model, receives a tool call, executes the tool, appends the result, and sends everything again. A median session in the traces used covers about 60 turns and grows to tens of thousands of tokens, all re-sent on every turn.

Answering a turn requires the KV cache for the full history. If the cache from the previous turn is still in GPU memory, only the new tokens need computing. If it was evicted while the agent was busy with its tool, the engine recomputes the whole history. Both engines have prefix caching, but neither knows that a request belongs to an agent that will return within a second. vLLM 0.33.0 accepts a session_id but treats it purely as metadata, with no pinning or time-to-live, and evicts blocks on a least-recently-used basis. SGLang 0.5.23 has real session endpoints, but its OpenAI-compatible chat API forwards only the session id, and the mode that keeps a session's KV cache pinned is disabled by default and requires the /generate API.

The experiment

To measure the cost, the author wrote a roughly 1,700-line Rust serving engine that keeps each agent's session in GPU memory between turns, shares the common prompt prefix across sessions, and admits new sessions only when those already resident still fit in memory. It was compared against vLLM and SGLang by replaying 24 real coding-agent sessions from the SWE-rebench-OpenHands dataset (791 turns) all running concurrently, using Qwen3-8B on a single NVIDIA L40S with an identical KV cache size of 173,200 tokens. Tool waits between turns were fixed at 0.5, 5 or 60 seconds. Measured coding-agent tool calls have a median gap of about 0.42 seconds, making the 0.5-second case the realistic one.

The numbers

With the 0.5-second wait, vLLM 0.33.0 finished all sessions in 2,264 and 2,222 seconds across two runs, recomputed 55% of history, and showed a median first-token time of 28.6 seconds. SGLang 0.5.23 with default LRU eviction took 2,013 and 2,048 seconds with 51% recomputation and a 35.0-second median; its TLRU option was similar at 52% and 36.3 seconds. The Rust session engine completed the workload in 1,506 and 1,586 seconds, recomputed only 8–16% of history, and produced a first token in 4.5 seconds, roughly five times sooner.

That works out to 1.31x faster than SGLang and 1.45x faster than vLLM overall. The advantage shrinks as tools get slower: at a 5-second wait it falls to 1.13x and 1.24x, and at 60 seconds all engines perform about the same, because few agents contend for the GPU at once.

What the study does not claim

The author is explicit about the limits. The prototype's decoding is 15–21% slower per token than vLLM's, so the entire gain comes from avoiding recomputation. The underlying idea is not new: InferCept, Pensieve and Continuum previously proposed keeping agent KV caches alive across tool calls. The test covers one model, one GPU, one load level and fixed tool waits, and did not evaluate CPU offload options such as LMCache or SGLang's HiCache, nor SGLang's streaming sessions, which might narrow the gap. Three of the prototype's pinning policies actually lost to vLLM at 60-second waits, since holding sessions too long blocks memory that other agents could use.

Why it matters

Anyone serving many concurrent agents on their own GPUs may be computing the same tokens twice without noticing. Both vLLM and SGLang can report cached tokens per request, through --enable-prompt-tokens-details in vLLM and --enable-cache-report in SGLang. If more than half of prompt traffic turns out to be recomputed history, as in this test, the workload rather than raw engine speed is the bottleneck. The author does not recommend switching tools: the Rust engine is a research prototype, and both projects already have open proposals for native session awareness, cited as vLLM PR #38514 and RFC #37003 and SGLang RFC #24656. The engine, the replay client, the raw results and an accompanying paper have been open-sourced in the agent-session-engine repository on GitHub.