· via dev.to (home feed)
NVIDIA Dynamo 1.6.2 switches oversized prompts to HTTP 400 and adds opt-in SGLang GC freeze
NVIDIA Dynamo 1.6.2 turns oversized TensorRT-LLM prompts into HTTP 400 client errors instead of HTTP 500, and adds an opt-in garbage collector freeze for SGLang workers to cut streaming stalls.
Oversized prompts now return HTTP 400
NVIDIA has shipped Dynamo 1.6.2, tagged 2026-11-05 as a patch release on top of v1.6.1, according to a dev.to write-up of the release notes. The fix with the broadest operational reach is narrow in scope: a prompt sent to the TensorRT-LLM backend that exceeds the engine's max_num_tokens limit now comes back as HTTP 400 with an invalid_request_error type, rather than the HTTP 500 api_error it produced before — and the worker process keeps running instead of being lost to the rejection.
The 500 behaviour was not new. The v1.6.0 release notes listed it as a known issue with no workaround, affecting non-streaming /v1/messages requests in particular. A request that could never succeed was being reported as a transient server failure.
Why the status code change matters
A 4xx tells a well-behaved client, SDK or gateway not to resend a request unchanged; a 5xx signals a server-side failure where a retry might succeed. Under a policy of three retries on 5xx, a single oversized prompt turns into four attempts against the frontend. The dev.to post runs an explicitly illustrative calculation — not a measurement — in which 50 oversized requests per second become 200 attempts per second, each one guaranteed to fail in the same way.
Observability pays a second price. Error dashboards and SLO burn alerts generally book 5xx as the serving team's failure and 4xx as the caller's, so a burst of oversized prompts used to surface as an api_error spike capable of paging on-call for something that is not an outage.
The post also points at where silent retries tend to live: default SDK retry behaviour, gateway and service-mesh policies, and queue consumers that requeue on any non-2xx response. Its advice is to log attempt counts per request id before upgrading, and to add a gateway-side pre-check that rejects prompts over the token budget before they occupy a GPU worker.
One caveat survives the fix. Apart from the oversized-prompt item, 1.6.2 inherits the v1.6.0 known issues, including one where the frontend returns a generic error body without the worker's message. Expect the right status class, not necessarily a useful body — and do not build client logic that parses error text.
An opt-in garbage collector freeze for SGLang workers
The second change targets Python garbage collection inside SGLang decode and aggregated workers. A new --freeze-gc-after-init flag, also settable via the DYN_SGL_FREEZE_GC_AFTER_INIT environment variable, ships disabled by default and is described in the release notes as a way to reduce streaming stalls in long-running workers, with a recommendation to check memory use against a control worker before enabling it broadly.
The dev.to author sketches the likely mechanism while stating they have not read the implementation. The general technique rests on standard CPython behaviour: the cyclic collector periodically walks tracked container objects, and a worker that has loaded a large amount of startup state gives every full collection a long walk — a pause on threads that may also be feeding tokens to open streams. CPython's gc.freeze() moves everything tracked at that moment into a permanent generation that later collections skip. The cost is that frozen objects are never collected, so cycles among them that later become garbage stay resident, which is why the control-worker comparison matters.
For rollout, the post suggests enabling the flag on a single canary decode worker per pool and comparing resident memory growth per hour, p99 inter-token latency and the count of streams with gaps beyond a stall threshold, over at least one full traffic cycle. It also argues that teams without an inter-token gap metric should add one first, since otherwise the flag's effect is invisible.
Image and dependency updates
Backend engine versions are unchanged from v1.6.1 — SGLang v0.5.21, TensorRT-LLM v1.3.0rc29 and vLLM v0.31.0 — with the churn sitting at the image level. The SGLang image moves from Ubuntu's Python 3.12.3 to a uv-installed CPython 3.12.16 with OpenSSL 3.5.9, a change the notes single out as workflow-affecting: without Ubuntu's libpython3.12 package, cuda-gdb inside that image no longer supports Python scripting. The planner image moves to Python 3.13.18 slim on a distroless runtime base. Elsewhere, Mooncake goes from 0.3.13.post1 to 0.3.14 in the SGLang and vLLM images, PyNvVideoCodec from 2.2.3 to 2.2.4, pyasn1 is constrained to at least 0.6.5 but below 0.7, diffusers is pinned at 0.40.0, and the libgnutls30t64 and libbrotli1 OS packages are refreshed.
Why it matters
For teams serving LLMs on Dynamo, this patch fixes a contract bug and adds a tuning knob rather than new features. Moving oversized prompts out of the 5xx bucket stops doomed retries at the first attempt and keeps a caller-side problem out of server-side error budgets. The GC freeze gives operators of long-lived SGLang workers a targeted lever against streaming stalls, provided it is canaried with memory watched closely. And the Python packaging shift in the SGLang image is a reminder that even patch releases can quietly break debugging runbooks.
- #nvidia
- #llm-serving
- #tensorrt-llm
- #sglang
- #inference