Pinning KV cache limits stops vLLM OOM crashes on GPUStack GPU fleets

What happened

A team running a four-node on-prem inference fleet, built from mixed RTX A6000 and RTX 4090 cards, moved off hand-rolled Docker launch scripts and onto GPUStack two months ago. According to a post on dev.to, the platform itself works as advertised: one control plane schedules models across whatever GPUs are available, starts a serving backend such as vLLM, SGLang or llama-box for each deployment, and exposes an OpenAI-compatible endpoint. But the team hit a failure mode the platform's placement logic did not anticipate, and it cost them a weekend.

Where the default estimate breaks

GPUStack decides where a model lands using a VRAM estimate derived from weight size plus a declared context length. As the author points out, that estimate has no visibility into activation memory peaks, CUDA graph capture, or the real distribution of prompts users send. Treating it as a runtime guarantee rather than a scheduling heuristic is what caused the incident.

The failure appeared with a 70B-parameter model in bf16, sharded across four cards with tensor parallelism set to 4 and the context length left at its default. The deployment looked healthy until real traffic arrived, with 6k to 8k token requests, at which point workers crashed with out-of-memory errors one after another and were restarted. It resembled scheduler instability but was ultimately a memory accounting problem.

The dev.to post walks through the arithmetic: Llama-3.3-70B consumes roughly 320 KB of KV cache per token, given 80 layers, 8 KV heads, a 128-dimension head size, K and V tensors, and 2 bytes per value. At a 32k context that is around 10.7 GB of KV cache per sequence, stacked on top of roughly 35 GB of weights per card. Two concurrent long sequences exhaust the card. The estimator had treated context as a fixed budget rather than a per-sequence allocation multiplied by concurrency.

The hardened configuration

The fix, per the author, is unglamorous: pin the values rather than accepting defaults. The team set max-model-len to 16384 after measuring their prompt p99 at 11k, capping the KV ceiling without doubling the pool. They set gpu-memory-utilization to 0.88, having found that 0.92 crashed with identical weights because graph capture under TP=4 does not fit the margin people typically assume it will. They capped max-num-seqs at 32 so the KV pool cannot be oversubscribed, enabled chunked prefill, and set swap-space to zero deliberately, on the grounds that host swapping converts an honest crash into a roughly 40x latency degradation that looks like a network fault at 2am. They also switched the KV cache to fp8, halving its footprint with no measurable regression on their eval subset, while cautioning that long-context retrieval degrades more than chat and should be tested first.

On a heterogeneous node where tensor parallelism was not possible, GPUStack fell back to llama-box with layer offload. There the equivalent discipline was capping GPU-resident layers at 60 of 80 on a 48 GB card; requesting all 80 produced a model that loaded and then died on the first 4k-token prompt.

Catching the slow decay

A second, subtler issue emerged: throughput dropped about 18% over 36 hours while GPU utilisation stayed flat. The author's diagnostic was to measure time-to-first-token percentiles and worker resident memory together. Monotonic RSS growth with p95 latency tracking it indicates KV pool growth and block fragmentation; rising p95 with flat RSS points at placement or preemption. Their fleet had the former, and it was resolved by the concurrency cap plus recycling workers based on an RSS-slope threshold instead of a fixed cron schedule.

One dilemma remains unsolved: a single endpoint serving both short-prompt, high-concurrency chat and a nightly batch job with 30k-token prompts. Splitting into two deployments doubles resident weights, while per-request routing between pools adds a hop the team has not budgeted to debug.

Why it matters

As orchestration layers like GPUStack make it trivially easy to deploy large models on existing hardware, the gap between a placement estimate and actual runtime memory behaviour becomes the operational risk. KV cache scales with sequence length times concurrency, which default settings neither measure nor bound. The practical lesson generalises beyond GPUStack: measure your prompt p99, pin context length, memory utilisation and concurrency, disable host swap, and watch TTFT alongside RSS to distinguish fragmentation from preemption before a slow decay becomes a 2am incident.