LangChain survey finds agent observability near-universal while evaluation lags far behind

Almost every team running AI agents can see what their agents did last week. Barely half can say whether the agents did it correctly. That is the central finding from LangChain's State of Agent Engineering report, which gathered 1,340 responses between 18 November and 2 December 2025 and was published on 23 May 2026. According to the survey, 89% of practitioners have implemented observability for their agents, rising to 94% among teams with agents in production. Systematic evaluation tells a different story: 52.4% run offline evaluations against test sets, 37.3% run online evaluations in production, and fewer than a third do both.

What the survey measured

The report puts overall agent production adoption at 57%, climbing to 67% at organisations with 10,000 or more employees. Full per-step tracing is in place for 62% of respondents overall and 71.5% of production teams, while online evaluation reaches only 44.8% of production teams. The same survey names quality — accuracy, relevance, consistency and tone — as the single biggest obstacle to getting agents into production, cited by roughly a third of respondents.

Those two findings sit in tension: the thing most likely to block a deployment is also the thing least likely to be measured. Around nine teams in ten can reconstruct what their agent did; about five in ten can judge whether it was any good.

Why dashboards miss wrong answers

A recent dev.to analysis argues the gap exists because tracing and evaluation measure fundamentally different things. A trace records that a tool call happened and that the API call succeeded. It cannot record whether the agent made the right call, because a confidently wrong chain of tool calls produces the same telemetry as a correct one.

The post illustrates this with two runs of a support agent that handles refunds. Their telemetry is practically identical: three spans, all marked OK, roughly 3.2 seconds, about 2,850 input tokens, a clean stop. Yet in one run the agent refunds the $40 order the customer complained about, while in the other it refunds a $2,300 order the customer never mentioned. The refund endpoint succeeds either way, so every span stays green.

The OpenTelemetry GenAI semantic conventions — the attribute set most observability platforms collect — cover the model used, token consumption, finish reasons and operation names. Every one of those is a property of the call rather than the answer. The conventions do define an evaluation namespace with attributes for scores and labels, but those fields remain empty until a team writes an evaluator that fills them. The practical consequence is that alerts wired to latency, token spend and error rate stay flat while quality degrades; from the runtime's point of view, nothing failed.

The established fix: check resulting state

Outcome measurement has a precedent that predates the current agent wave. Sierra's τ-bench, a benchmark for tool-using agents, scores conversations by comparing the state of the database at the end of a task against a pre-annotated goal state — not the wording of replies and not which tools were named. Applied to the refund example, a short function that diffs the database before and after the run, checking which orders were refunded and for how much, catches the mistake instantly, even though no telemetry field distinguishes the two runs.

Offline evaluations on curated test sets catch such failures before release; online evaluations catch them while traffic is live. Most teams, per LangChain's numbers, do neither consistently.

Why it matters

Observability was adopted quickly because vendors sell it as a drop-in integration. Evaluation cannot be bought the same way, since it requires each team to define what a correct outcome means in its own domain — a product decision, not an installation step. Teams that skip both offline test sets and production checks have no signal that moves when their agents start doing the wrong thing competently. Latency will not regress and error rates will stay at zero; the first alert usually arrives as a customer complaint.