Vercel explains how to build end-to-end evals for AI agents that use your software

Vercel has published a practical guide to evals for AI agents, aimed at teams building either the agents themselves or the software agents consume — APIs, CLIs, SDKs and MCP servers. The post argues that agents now pass through the same funnel a human user does — discovery, selection, access and success — and that evals are the way to measure the final stage: whether the agent actually completes the user's task correctly.

That turns out to be harder to check than it sounds. According to Vercel, an agent can technically complete a request while failing the user, ordering a hot latte when an iced one was asked for. The request succeeds, so observability logs look healthy. And because LLM output varies between runs, a single passing manual test says nothing about whether a fix actually worked. You need to repeat the same task, grade the outcome and aggregate results before and after any change.

The agentic testing pyramid

Vercel organizes agentic tests by how many LLM calls they require: deterministic tests with none, one-turn evals with one, and end-to-end evals spanning a full task. That count, the company says, has proven a good proxy for the effort, runtime and cost of each test, so the cheap and fast tiers should form the bulk of a suite — an agent-flavored extension of the classic testing pyramid. The rule of thumb is to start from the problem you want to catch and pick the lowest tier that can catch it.

One-turn evals check a single decision, such as which command the agent runs first. But success depends on the entire sequence of steps, and Vercel notes that each step can look reasonable in isolation while the final outcome is still wrong. That is why the Success stage calls for E2E evals — and the post points out that well-known coding benchmarks like SWE-bench and Terminal-Bench already fit this pattern.

A coffee-ordering worked example

The guide builds its example around a fictional coffee-ordering API, SF Beans, exposed to agents through a CLI called sfbeans. A typical customer request asks for a large iced latte with whole milk, ready for pickup by 9:00 a.m. and under $8 including tip, in San Francisco's Mission district. Of four nearby cafes in the sample data, three each violate one constraint — no iced drinks available, over budget once tax and tip are added, pickup too late — leaving exactly one valid choice. Only a full-task evaluation can confirm the agent lands on it.

Vercel implements the eval with Harbor, an open-source framework from the creators of Terminal-Bench that runs agents in containers, records their actions and checks results. The post defines its vocabulary: a task is one problem with a starting environment and a success check; a variant is the agent setup under test, such as the model, prompt, tools, memory and skills; a trial is one attempt by one variant; a run covers every variant on every task; and a trace records everything that happened during a trial.

Because full coverage of everything an LLM might do with a product is impractical to even delimit, the guide recommends starting with one narrow, user-relevant goal, found through observability data, customer support tickets and churn, or the product roadmap. In the example, the top support complaint — agents ordering the wrong drink — sets the goal.

Choosing the right metric

Vercel's core metrics are success rate, pass@k (the chance at least one of k trials passes, answering whether the agent can do the task at all) and pass^k (the chance all k trials pass, answering whether it can be relied on every time). Efficiency measures — steps, time and dollar cost — are counted only on successful trials.

In the sample run, Claude Code paired with Anthropic's Opus 5.5 completed the iced-latte task in three of five trials: a 60% success rate, roughly 99% pass@5 but only about 8% pass^5. Both figures are correct and simply answer different questions. For a daily coffee order, where a failure means the user paid for the wrong drink with no retry, Vercel says sfbeans would optimize for pass^k; products where a failed attempt is cheap and caught early can settle for pass@k.

The post closes its metric discussion with two cautions: overall averages can hide individual tasks that fail often, so results should be inspected per task, and a handful of flawless trials means no failure has been observed yet, not that failure is impossible.

Why it matters

Agents are becoming a genuine distribution channel — Vercel's example detects agent traffic from tools like Claude Code and Codex via user-agent headers — and builders of the APIs and CLIs those agents use need a testing discipline designed for non-deterministic users. The guide offers a transferable recipe: use the pyramid to place tests cheaply, run full-task evals in a framework like Harbor, and pick the metric — pass@k or pass^k — that matches what a failure actually costs your user.