Hosted LLM endpoints may silently cap context at 32K regardless of model card, developer reports

What the post claims

A developer writing on dev.to reports that hosted endpoints for open-weight models frequently deliver a much smaller context window than their model cards advertise. The author, who builds Grunz, a hosted chat and coding agent, says that across the endpoints tested, most served about 32K tokens, a handful reached 256K, and none actually delivered the 1M-token figures that appear in release announcements.

The distinction the post draws is between the context length the weights support and the length an operator chooses to serve. The author considers the capping itself defensible: KV-cache memory grows with context length, and offering maximum-length contexts to every request would be unsustainable for a provider balancing concurrency. The problem, as the post frames it, is that the effective limit is almost never documented and that exceeding it produces no error at all.

Truncation that never announces itself

According to the post, an oversized request does not return a 413 or a warning. It is served against whichever ceiling is lowest, and the beginning of the context is dropped without notice. In a chat product this reads as the model gradually becoming less coherent over a long conversation, which most people would attribute to the model itself.

For agents, the post describes a more damaging pattern: a long build run reaches the ceiling, compaction kicks in, the original goal statement gets summarised into something vague, and the model then works from its own compressed notes, loses precision about what it was doing, and restarts its plan without signalling anything. The visible symptom is a model that appears bad at long tasks; the actual cause is the serving configuration.

Detecting the real ceiling

Neither provider documentation nor model cards were reliable for this in the author's experience, and the post offers two manual workarounds. The first is to send a deliberately oversized prompt, for example 500K tokens against an endpoint that claims 1M, and inspect the error text, which often reveals the true maximum even when the docs omit it. The second is to locate the point in a long run where the earliest content stops influencing the output, then count backwards to estimate where truncation begins.

Knock-on effects

The post draws out three consequences. Benchmarks, it argues, measure a model at one particular served context length, not the model in the abstract, since the same weights behave differently at 32K versus 200K. Comparing providers on price per token without pinning the served ceiling is therefore not a like-for-like comparison.

Second, gateways with fallback routing can shift the ceiling mid-session. A run that begins on a 200K provider and fails over to a 32K one will not error, it will truncate, turning a capacity detail into a silent correctness problem that nothing in the response exposes.

Third, RAG pipelines tuned against an assumed budget from the model card can be quietly invalidated. If retrieval parameters such as chunk size, top-k and reranking were tuned for 128K but only 32K arrives, the system over-retrieves, top chunks get cut, and answers arrive looking authoritative while being incorrect, prompting teams to re-tune embedding models that were never at fault.

Why it matters

The findings come from one operator's testing rather than a systematic survey, but they point at a verifiable gap between advertised and served context that anyone building on hosted open-weight models can and should check. The author's closing request is for providers to expose the effective served context per request, through model metadata or a response header, instead of leaving every client to reverse-engineer it from error strings. Until something like that exists, measuring the actual ceiling before benchmarking, tuning retrieval or comparing prices is a cheap safeguard against failures that look like everything except what they are.