Vercel Sandbox now runs Terminal-Bench and other Harbor evals in Firecracker microVMs

Vercel Sandbox can now host Harbor evaluations, Vercel announced on its blog, giving teams a way to run coding-agent benchmarks such as Terminal-Bench in isolated cloud microVMs instead of on local hardware.

Harbor is the open-source harness behind Terminal-Bench, and according to Vercel its registry also covers benchmarks such as SWE-bench, tau3-bench and OSWorld. The integration comes down to a single flag: pass --env vercel to harbor run and every trial runs in its own Firecracker microVM. Because trials no longer share a machine, Vercel says evaluations can be parallelized well past the limits of a local computer.

What the integration does

Each trial gets its own Firecracker microVM, which fits what agent benchmarks typically want: every task starts from a clean, isolated environment, and one failing trial cannot disturb another. Concurrency is set through the --n-concurrent flag, and Vercel's worked example runs eight trials at once against the terminal-bench-2-1 dataset using the fx agent.

Networking and credentials stay outside the VM

Two design choices stand out in how Vercel handles isolation. First, a task's network policy is enforced at the sandbox firewall, outside the VM, rather than by anything running inside it. Second, an optional credential injection feature attaches secrets to matching outbound requests at that same firewall, so the credentials never reach the sandbox itself.

The second point matters for this use case specifically. An agent under evaluation executes arbitrary commands inside its environment, and an API key stored in the VM is something that agent could read or leak. Keeping secrets at the firewall layer removes that exposure while still allowing authenticated requests, such as calls to a model provider, to pass through.

One key, many models

The feature is paired with Vercel's AI Gateway. A single AI_GATEWAY_API_KEY provides access to hundreds of models across multiple providers, and benchmarking a different model is the same command with a different --model value. In its example, Vercel swaps an Anthropic model for an OpenAI one with a one-flag change and reruns the same benchmark.

Getting started

The integration requires Harbor 0.22.0 or later. Setup involves installing Harbor with the Vercel extra using uv, exporting a VERCEL_TOKEN and an AI_GATEWAY_API_KEY, then invoking harbor run with the terminal-bench dataset, an agent, a model, the Vercel environment flag and a concurrency setting. Vercel has published a step-by-step guide along with Sandbox documentation covering setup, configuration and troubleshooting.

Why it matters

Evaluating agents has quietly turned into an infrastructure problem. Benchmarks like Terminal-Bench require booting a real environment, letting an agent work to completion and scoring the outcome, often across many trials and multiple models. Running that locally bounds the experiment by one machine's resources and places credentials within reach of the very software being tested.

Moving those runs to disposable microVMs addresses both constraints: concurrency scales with the cloud rather than a laptop, and secrets stay outside the sandbox. Combined with a single gateway key that abstracts away model providers, comparing agents and models across a whole benchmark suite becomes a routine command rather than a bespoke setup, which appears to be exactly the friction Vercel is aiming to remove.