Long-context LLMs start failing well below their advertised token limits

What the article claims

According to a developer walkthrough published on dev.to, large language models begin to fail on long-context tasks at input sizes far below their advertised context windows, and past a certain point additional context makes answers worse rather than better. The core argument is that a model's stated context length describes how much text fits into a prompt, not how much text the model can actually use effectively.

The window is a payload limit, not memory

The piece opens by dismantling a common mental model. LLM APIs are stateless: nothing persists between calls, so every request re-sends the entire conversation, including the system prompt and every prior question and answer. The context window only defines how much text the provider will accept before rejecting the request, and everything in that payload gets processed on every call.

Attention scales quadratically

Inside the model, attention lets each token weigh every other token, which means the number of token pairs grows with the square of the input length. The article works through the arithmetic: 10 tokens produce 100 pairs, 1,000 tokens produce a million, and 100,000 tokens produce 10 billion. Ten times the context is roughly a hundred times the work. Key-value caching explains why the first response in a long chat feels slower than later ones — but the cache saves compute, the author notes, and does nothing to protect answer quality.

Three mechanisms behind the degradation

The first mechanism is that attention is a fixed budget. Softmax forces each token's attention weights to sum to one, so pasting in fifty more documents does not give the model more focus; it divides the same focus across more material. The author likens it to file search: a query against ten files is trivial, while the same query against ten thousand pushes the right result deep into the results — except the model never shows you that results page, it just produces an answer.

The second is positional. Citing the "Lost in the Middle" research by Liu et al., the article reports that when researchers moved a single correct answer around inside an otherwise identical long prompt, accuracy stayed high near the beginning and the end but dropped sharply in the middle. A critical paragraph sitting around the 60 percent mark of a long prompt occupies the worst possible position.

The third is distractors in real data. Needle-in-a-haystack evaluations are flattering because the needle looks nothing like the hay. Production corpora are different: they contain version two and version three of a document, an old changelog, and a chat thread that contradicts all of them — several plausible chunks where only one is correct, and the model cannot tell them apart. Accuracy declines steadily as input grows, even on tasks the same model handles well at short lengths.

What developers can change

The article closes with four concrete adjustments:

• Send less, but better. In retrieval pipelines, search broadly, re-rank the results, then pass only the top three to five chunks instead of fifty.
• Exploit the edges. Put instructions at the top, data in the middle, and repeat the actual question at the bottom immediately before the model answers.
• Stop reusing one giant thread. Split a job into steps, give each step a fresh, small context containing only what it needs, and pass short summaries between steps — which, per the author, is largely what agent frameworks do under the hood.
• Measure on your own data. Run roughly twenty real questions from your application with 2,000 tokens of context, then the same questions with 20,000, and compare the answers side by side to find your own effective limit — which will likely sit below the vendor's specification.

Why it matters

For anyone building RAG systems or long-context pipelines, the advertised window is a ceiling on input size, not a target to fill. Pushing toward it carries three simultaneous penalties: quadratic attention cost, slower first responses, and measurably worse accuracy. The practical takeaway is to treat context as expensive bandwidth — retrieval quality and prompt structure buy more than raw capacity, and a team's usable context length has to be measured empirically on its own data rather than assumed from spec sheets.