Open-source Gauge CLI runs coding agents against your docs to find friction

Gauge, a new open-source command-line tool from the gauge-sh project, appeared on Hacker News on November 5, 2026, targeting a gap many maintainers now recognise: making products usable not just by humans but by coding agents. According to the project's GitHub repository, Gauge runs real agent sessions against your docs, CLIs, SDKs, MCP servers or agent Skills, records where the agents struggle, and then lets your own coding agent test candidate fixes before you publish anything.

A measurement loop for agent usability

The project describes its method as hillclimbing: change something, measure whether the change helped, and iterate. The loop runs in stages. You first define realistic tasks as evals or preference prompts, each with a stable identity and observable success criteria. Gauge then launches fresh Docker sandboxes, injects the working-tree version of your configured inputs, and runs each task with a real agent. Afterwards, the session evidence is examined to identify what slowed down or blocked the task, you or your agent edits the product or its guidance, and the affected cases are rerun to see whether the friction disappeared without introducing new problems.

Gauge currently supports four agent harnesses: Claude Code, Codex, Pi and OpenCode, using either native logins or API keys. Agent usage is billed through your existing provider or subscription. The tool normalises events from all four harnesses into consistent transcripts, which makes cross-agent comparisons possible.

The kind of problems it surfaces

The repository lists typical findings from real sessions:

• an interactive command halts an agent that is waiting on input it will never receive
• an agent reaches for deprecated commands or outdated API patterns
• a Skill sits unused even though it would have helped with the task
• an agent confidently claims a limitation that does not exist
• an agent cannot piece together an intended workflow that spans multiple commands

The right fix may live in error messages, help text, documentation, Skill content or the code itself. For an npm CLI or SDK, you build and pack a tarball, and Gauge installs that pre-release artifact inside the test sessions, so a fix can be validated before it ships.

Local-first setup

Local mode needs Node.js 22.13 or later, Docker running Linux containers, and credentials for the agent under test. Installation is a global npm install of @withgauge/cli, followed by a command that pulls a Docker image pinned to your CLI release, containing the agent harnesses and a set of common development tools.

Initialising a project creates a private .gauge/ workspace holding a SQLite database of measurements and analyses; a plain init touches no tracked files. An optional --cases flag adds shareable files — a gauge.json manifest, a gauge-cases/ directory and guidance appended to AGENTS.md — which the project says should be committed for teammates and CI use.

In local mode, the analysis work falls to your own coding agent: it reads the transcripts, judges each criterion and publishes structured findings through a CLI command. A compare command then summarises completion rates, turns, duration and criterion verdicts between a baseline batch and a candidate batch, so you can see whether an edit actually helped. No Gauge account is required for any of this. Hosted Gauge, the paid counterpart, additionally supports docs, authenticated MCPs and authenticated APIs as inputs, and performs the session analysis automatically.

Why it matters

Coding agents are becoming a primary interface to software, and whether an agent can discover the right command, understand an error message or infer an intended workflow increasingly determines whether a product gets used at all. Human-centred documentation review does not capture any of that. Gauge's bet is that agents can test for agents: run the session, measure the friction, iterate on the artifact. By open-sourcing the loop and keeping it runnable locally with Docker and existing agent subscriptions, the project lowers the barrier for CLI, SDK and docs maintainers who want evidence — not intuition — that their latest change actually made things easier for Claude Code or Codex. Whether agent-experience testing becomes as routine as CI remains an open question, but Gauge is an early attempt to productise that workflow end to end.