Test across 4,900 LLM sessions shows typos are harmless but missing quote marks break prompts

A controlled experiment published on dev.to by Vadim Albarov concludes that anyone polishing the spelling of their prompts is fixing the wrong thing. Across roughly 4,900 model sessions, frontier LLMs scored almost identically on clean text and on prompts with 70% of the words misspelled — while a single wrong punctuation mark cost the same models 8 to 23 percentage points.

What was tested

The author built two prompt sets. The easy set had 12 one-line tasks with short, checkable answers, each written at five error levels from clean to heavily garbled. The hard set had nine prompts of 200 to 450 words where the answer depended on a buried detail, and it separated the two kinds of damage. Spelling noise: a script misspelled 35% or 70% of words while never touching the ones the answer depended on. Structural breaks: spelling stayed perfect, but exactly one meaning-carrying mark was wrong — a missing closing quote, a colon dropped before a list, or "9.45" written where "9:45" was meant.

Two realistic profiles were layered on top: non-native grammar, and simulated voice-to-text with no punctuation, no capitals and numbers spelled out. The model lineup spanned Claude Opus 5, Opus 5.5, Sonnet 5, Haiku 4.5 and Fable 5.1 at low and medium effort, plus seven local models served through Ollama.

Spelling was free, structure was not

On the hard set, the frontier models all but matched their clean-text scores under 70% misspelling — Opus 5.5 dipped a single point — and all scored 100% under the non-native grammar profile. A shopping-list arithmetic prompt reduced to fragments like "3dolar" and "2notbook" still returned correct answers. But one structural break with flawless spelling hurt every frontier model: Opus 5 and Haiku fell from 100 to 88, Sonnet from 100 to 77.

Small local models had rougher baselines and rougher drops. Mistral 7B went from 52 on clean prompts to 26 under heavy typos, confirming, per the write-up, that frontier models absorb spelling noise smaller models simply cannot.

Which breaks mattered

Most punctuation errors were recovered from context. European-style decimals, a time written as "9.45", a shifted apostrophe in "managers", and a dropped comma in an ambiguous ticket-filtering instruction were all handled by every Claude model tested.

Two breaks were not. The missing closing quote was the worst: asked to count a word's occurrences inside a quoted paragraph, every model counted three extra occurrences in the text after the paragraph and answered 8 instead of 5. According to the post, Opus 5.5 explicitly noted that the closing quote was missing and still counted everything — it diagnosed the break and obeyed it anyway. The five frontier models scored between 0 and 33 on that task.

The dropped colon broke the boundary between instruction and data. A prompt asking for the plural forms of a word list, with no colon marking where the list began, made "reply" read as one more word to pluralize. Sonnet and every local model took the bait.

Dictation is a different problem

The simulated voice-to-text profile is where small models collapsed. gemma4 fell from 78 to 44 and granite 4.2 from 93 to 56, and every local model scored zero on a dictated list task, done in by the combination of spelled-out numbers and the missing list colon.

Frontier models mostly coped, with honest exceptions: Sonnet read "two fifty" as $250 in six trials out of six and answered 375,000 instead of 3,750. As the author notes, spoken numbers are genuinely ambiguous, so the model has to guess.

What didn't help

Raising effort from low to medium changed nothing on any task, which the author attributes to structural breaks not being reasoning problems — more thinking does not restore a lost delimiter. Prepending a warning that the message may contain mistakes had zero effect on three of the five frontier models and mixed or negative effects on the local ones.

Format also degraded before accuracy did. Under heavy noise, Haiku drifted into unprompted explanations and, at the heaviest error level, refused a nonsense-word logic question four times out of eight, having answered the same question fine with clean grammar.

Why it matters

For anyone building LLM features, the lesson is to stop treating prompt hygiene as proofreading and start treating it as parsing. Spelling and grammar can be ignored; the delimiters that separate instructions from data — closing quotes, colons before lists, commas around "except" and "not" — cannot. That applies doubly to transcription pipelines, where dictation strips punctuation and makes numbers ambiguous, and to deployments running 7B to 14B local models, which the data suggests need clean delimiters to hold up at all.

The caveats are real: three trials per cell, hand-simulated dictation rather than real speech-engine output, and a minimal system prompt. But the harness, prompts and raw replies are published openly, so teams can rerun the test on their own tasks before deciding which characters in their templates actually carry weight.