Coding agents prefer grep to LSP tools until results include inline source

A small study on how coding agents retrieve code context found that models overwhelmingly prefer grep over LSP-backed semantic navigation on routine tasks, and that forcing the semantic path can make them worse at the job. The conclusion is not that lexical search is superior, but that a tool's usefulness to an LLM depends as much on the shape of its output as on the precision of its results.

The study, published as a blog post on AgentConnect and surfaced on the Hacker News front page, compared two retrieval interfaces given to coding agents: plain grep, which matches text, and LSP-backed tools that navigate definitions, references and document symbols, and can therefore distinguish a real function call from the same word in a comment. The pilot ran three Claude models — Opus 4.8, Sonnet 4.6 and Haiku 4.5 — across several Python and TypeScript repositories. Token use was measured only when both approaches completed a task successfully, which the author notes controls for a common evaluation error: failed runs can look efficient simply because they stopped early.

Task type drove the routing

On simple code-location tasks, the models chose the semantic tool only 0% to 6% of the time when both were available. Forcing a semantic-first path cut success in that arm from 100% to 89%. Reference-completeness work — finding every caller of a symbol — flipped the pattern: models reached for LSP navigation 45% to 57% of the time without prompting.

Even there, the picture was mixed. According to the study, the LSP path reached precision of 1.00 versus 0.76 for grep by eliminating false matches, but recall stayed around 0.66 in both arms. Semantic navigation did not surface more true calls; the remaining limit was how thoroughly the agent worked, not retrieval precision. For the stronger models, the precision gain also came with higher token use rather than a saving.

The codebase mattered as much as the task. On the clean TypeScript repository remeda, LSP navigation produced no F1 gain and used 16% more tokens. On the noisy TypeScript repository hono, it improved F1 by 0.246 and used 12% fewer tokens. The useful predictor of whether semantic navigation pays off was how badly grep's precision degrades on a given codebase — lexical noise — not whether the language is statically typed.

Output shape was decisive

The most striking result came from a change to the LSP tools' response format. The tested tools initially returned only a location — file path, line and column — forcing the agent to open the file to inspect the code, while grep returned the matching line immediately. The author modified the semantic response to attach roughly two lines of surrounding source to every reference, leaving the semantic backend and the set of references unchanged.

That change alone lifted pass@1 on multi-file rename tasks from 0.67 to 0.83 and cut follow-up file reads from 15.2 to 3.2 per episode — below grep's own 4.3. The study connects this to guidance Anthropic has published on writing effective tools for agents: tools are interfaces for non-deterministic agents, so the context they return is part of the design, and a semantically correct tool can still create a poor workflow if each result requires several extra actions to interpret.

Two explanations, one proven

The author offers two explanations for grep's advantage. The structural one follows directly from what the tools retrieve: a rename may need to touch comments, docstrings, configuration or strings, which find_references deliberately does not return while grep can. For text-wide edits, grep is the better retrieval tool regardless of how well trained the model is on LSP navigation.

The distributional explanation is a hypothesis: models may have learned concrete action patterns — prompt, tool call, readable result, next action — during training, and favor familiar loops. The study did not manipulate training data, the author stresses, so this remains unproven, and the output-format improvement is consistent with it rather than evidence for it.

Why it matters

For teams building agent tooling, the takeaway is that retrieval precision is not the metric that determines adoption. The amount of context returned per call, and the number of follow-up actions needed to interpret it, decided whether the agents used a tool effectively — a single change to the response payload outperformed any property of the retrieval backend.

The study frames the harness — the instructions in context, the available tools, their names, input schemas, the shape of their results and errors, and the loop deciding what the model sees next — as part of the capability surface, co-equal with the model itself. A model and its familiar tool loop act as one unit, so teams shipping smarter retrieval tools that agents ignore may not have a model problem but an interface problem.

The author is careful to present the results as conditional rather than categorical: agents routed by task, and the value of semantic navigation varied by repository. This is not a verdict against the LSP, whose protocol covers far more than navigation, but a warning against assuming that better retrieval semantics will be adopted as-is.