NVIDIA OpenShell blocked agent actions but its audit export logged zero records, evaluation finds

An independent evaluation of NVIDIA's OpenShell, a sandbox runtime for AI agents, found that enforcement worked while the evidence trail did not: across two runs, the documented OCSF JSON audit export came back completely empty. The measurements, along with an open-source tool the author built to address the gap, were published in a post on dev.to.

What the test measured

According to the write-up, the author ran OpenShell 0.1.5 on a GitHub Actions runner with the Docker driver, enabled the export option exactly as NVIDIA's documentation instructs, and executed real commands inside the sandbox. The environment reached its ready state and the commands ran. A denied DNS lookup did trigger a native shorthand event — NET:REFUSE, marked DENIED with the reason policy_dns_ineligible — but it appeared only on the VM console.

The documented OCSF-JSON export, by contrast, produced zero records in both runs. Because the supervisor's file sink sits on a tmpfs that external tools cannot read, per NVIDIA's own documentation, the internal sink could not be inspected out of band either. The author's conclusion is that, in the current build, the runtime exports nothing an outside evidence system could consume. An earlier phase of the evaluation also reported that a blocked DNS query left no event at all.

The response: a runtime-agnostic evidence layer

To close the gap, the author released Agent Seatbelt, an Apache-2.0 licensed layer designed to sit above any agent runtime rather than replace one. Its main pieces:

• Deterministic policy at the boundary. Tool calls, network egress, DNS, file access, budgets and credential grants are decided by OPA/Rego policies running outside the agent process, with no model involved in the decision path.
• An append-only, hash-chained ledger. Every allow and deny becomes a record stamped with the hash of the exact policy pack that produced it; an offline verifier can identify the specific record if a single byte is altered.
• Coverage accounting. Each adapter publishes a manifest and, after a run, the ledger is reconciled against independent ground truth, so any action lacking a matching record causes the run to fail. The system verifies coverage rather than taking it on faith.
• Scoped credentials. A broker grants secrets per run, per action and per target; the model never sees raw values, and the ledger logs the grant or refusal, never the secret itself.

Reported results and stated limits

The author reports phase-1 figures, first measured against a reconstructed OCSF stream: coverage reconciliation of 11 out of 11 actions on a clean run and 13 out of 13 with planted violations; five planted attacks — DNS secret exfiltration, a denied tool call, a budget-cap breach, an out-of-scope file write and credential abuse — all blocked and recorded; and zero false blocks on the clean run. Decision latency sat at a p95 of 27.5 milliseconds against a 50-millisecond budget, and a one-byte tamper caused the verifier to fail and name the altered record. The harness even caught a bug in the author's own sample code, where a client silently bypassed the proxy via a no_proxy setting and left one fetch unrecorded.

The post is also explicit about limits. The original blocked-UDP-DNS case could not be re-run on the vendor's image because the default sandbox ships without dig, curl or python3, and no authenticated live Claude Code session has been run yet, only the hook protocol path. On the runner where OpenShell's export came back empty, the Seatbelt side passed 13 of 13 against ground truth, the author says.

Where the author concedes others are ahead

The write-up acknowledges that OpenShell and a project called nono provide the actual isolation Seatbelt deliberately lacks, that nono shipped hash-chained audit logs first within its own sandbox, and that Infisical's Agent Vault is a more mature commercial credential broker than Seatbelt's in-memory phase-1 equivalent. The narrower claim is unifying policy, evidence, credential grants and verified coverage on one layer that works across runtimes.

Why it matters

Sandbox enforcement without records answers only the easy half of agent security. The hard half — proving, after an incident, exactly what an agent did and what was stopped — depends on audit trails that survive independently of the runtime. If a major vendor's sandbox can block correctly while its documented export silently writes nothing, teams running agents in production face an accountability gap that no amount of blocking closes. These findings rest on one developer's measurements of a 0.1.5 release and the project itself is early stage, but they surface a question every agent deployment should be able to answer: if you were audited tomorrow, could your sandbox show its work?