Vercel's agent-desktop lets coding agents drive native apps via accessibility trees

Vercel ships agent-desktop

Vercel Labs has released agent-desktop, an open-source tool that lets coding agents read and operate native desktop applications through the operating system's accessibility tree, without moving the user's pointer or stealing focus.

According to a write-up on dev.to, version 1.0.2 shipped on October 30. The README pitches it as "computer use for AI agents" and describes a Rust CLI plus a C library, licensed Apache-2.0, running on macOS, Windows and Linux. It contains no model: an agent drives it entirely by running shell commands, installing it with npm like any other CLI.

Vercel's Chris Tate reportedly described it as "agent-browser, for your whole computer," referencing Vercel's earlier agent-browser tool for web pages, and CEO Guillermo Rauch shared the launch with the line "An agent on every desktop."

What shipped

The dev.to post, citing ShiftDelete, says the project began in February and later moved under the Vercel Labs organisation. Windows 10 1809+ support arrived in v0.10.0 on October 18, and v1.0.2 added a documentation site and made AGENTS.md the single instructions file for agents. MarkTechPost and AlexTech covered the launch the following day, and the repository had roughly 6,900 stars as of November 2.

The documented floor is macOS 13 or later (Intel or Apple Silicon) via the AX API, Windows 10 1809 or later via UI Automation, and Ubuntu 24.04 or later via AT-SPI, on both x64 and ARM64.

How the loop works

agent-desktop does one step per call: each command prints a single JSON envelope and exits, so the observe-act-verify loop lives in the calling agent. A typical sequence snapshots an app's interactive elements (each returned with a reference ID), acts on one of those references, and snapshots again to confirm the result.

References are tied to the snapshot that produced them. Since v0.10.0 removed bare refs, old-format references fail with INVALID_ARGS; if the UI has changed underneath the agent, the call returns STALE_REF and the agent should re-snapshot. Multiple matching windows produce AMBIGUOUS_TARGET along with candidate window IDs. Every error carries a disposition, and the README advises retrying only when the retry disposition is marked safe.

Calls are headless by default: reference-based actions go through the accessibility API without moving the pointer or taking focus. A global --headed flag permits physical mouse and keyboard input when an app genuinely requires it, and ShiftDelete reports that when multiple agents run simultaneously, each gets its own coloured overlay cursor and physical-pointer actions take turns.

The token claims are self-reported

The project's headline argument against screenshot-based computer use is cost, but the dev.to author stresses that every figure so far comes from the project itself, and admits not having run the tool.

The builder reported 273 tokens to snapshot one region of Apple Maps versus 3,811 for the whole window, and a simulated 52-step task dropping from 198k to 31k tokens, as summarised by CellCog. AlexTech puts the savings from the "skeleton" mode (map an app shallowly, then drill into one subtree with --root) at 78 to 96 percent against full-tree dumps in Slack, VS Code and Notion, while explicitly flagging these as unverified self-reported figures. CellCog adds that no shared yardstick exists for such comparisons yet.

Limits, permissions and privacy

The documentation is blunt about constraints. The tool reads accessibility trees and does not match pixels, so any UI that publishes no accessibility tree is invisible to it — effectiveness depends on how well each app was built. It does not automate web pages, which remain agent-browser's job, though it can open a DevTools port for web content inside Chromium or Electron apps. The CLI exposes 60 command names across platforms, four low-level input commands fail closed everywhere, and a successful action does not prove the UI changed, which is why the docs recommend a follow-up read such as wait --element with a value predicate.

On macOS, the Accessibility permission goes to whatever application runs agent-desktop — if the agent lives in a terminal, the terminal is what asks — and the README requires TMPDIR to be user-owned with mode 0700. The README also warns that snapshots, clipboard reads, screenshots and debug artifacts can contain private desktop content, so agent transcripts may now hold whatever was on screen. The FAQ goes further, noting that redaction "does not make a screenshot safe to publish." The native CLI requires no remote model service, although an optional JEV helper does send goals and screen descriptions to a configured model provider.

Why it matters

Agent-driven computer use has so far largely meant screenshot-based control: expensive in tokens, brittle against UI changes, and disruptive because it hijacks the physical pointer. By routing through accessibility trees instead, agent-desktop offers a structured, cheaper path that keeps the user's machine usable while the agent works, and it does so as a plain CLI that any existing coding agent can shell out to, with no bundled model.

The caveats matter just as much. Token savings are self-reported with no common benchmark, and apps with poor accessibility support stay out of reach entirely. As the dev.to author notes, a click in Finder leaves no git diff: review workflows built around file changes cannot see what an agent did in another application. Teams adopting this will need new review and redaction habits, not just a new tool.