Tesla FSD crash and Anthropic approval data expose the limits of human supervision of AI

A Tesla struck a patrol car with the driver reportedly asleep

On 13 October at around 11:45 pm, a Tesla drove through a closed-off lane on Interstate 80 in California and hit a highway patrol car that had stopped at an earlier crash. Two officers were injured. According to a dev.to article by Jules Robineau, which cites reporting from Electrek and CBS Sacramento, the 22-year-old driver told police he had been asleep and that the car was running Full Self-Driving. The police have not confirmed that FSD was actually engaged at the time of the crash.

The product's full name is "Full Self-Driving (Supervised)", a label Tesla adopted in late March 2024 when it retired the "FSD Beta" name. Tesla's manual requires a fully attentive driver ready to intervene at any moment, and a cabin camera monitors attention. The company's own support page warns drivers against complacency while also claiming seven times fewer collisions with FSD engaged — a Tesla figure based on the company's own data.

A human is the wrong component for this job

The article connects the crash to a long-standing finding in automation research. In 1983, psychologist Lisanne Bainbridge published "Ironies of Automation", arguing that automation tends to leave humans with the one task they perform worst: watching a machine and stepping in when it fails. Studies from the 1950s onward found that people cannot sustain effective attention for much more than half an hour when almost nothing happens, and the longer a system works correctly, the less anyone monitors it.

Road data appears to bear this out. According to the article, NHTSA has been running an in-depth investigation since March covering 3.2 million vehicles, focused on crashes in reduced visibility. In the cases reviewed, the system warned the driver only at the very last moment, and the agency found drivers may not have realised they were expected to intervene. A 2024 IIHS study that followed 14 Tesla drivers for four weeks found first attention reminders rose 26% while stronger alerts fell 64%, meaning drivers had learned that a quick nudge on the wheel silences the car. Informal videos also show the cabin camera being fooled by a folded bill or a doll's head.

The same decay shows up in coding agents

The piece then extends the argument to developer tooling, where permission prompts mirror the FSD bargain: the machine acts, the human approves. In August, Anthropic published figures showing that Claude Code users approve 97% of permission prompts. In a test with 1,053 paid testers, each of whom had a dangerous command slipped into their session, participants caught only 13.6%. The breakdown is more telling: about 17% of dangerous commands were blocked early in a session, but only about 5% after 50 prompts — the same vigilance wear Bainbridge described, measured on developers. The article adds a caveat: Anthropic also sells an automatic mode that replaces these approvals, so the numbers serve its product, though they are the only dataset of that size the author found. The 2026 Stack Overflow survey adds contrast, with only 6.6% of developers saying they trust AI for an important decision.

Design as if nobody is watching

The lesson the author draws is not that humans are careless but that vigilance cannot be the safety mechanism. Forbidden actions should be blocked by a rule in code, not by a prompt or an instruction. Human approval should be reserved for genuinely irreversible actions — deleting, sending, paying, deploying — so that each request is rare enough to actually get read. Every agent action should leave a trace that can be audited later with a clear head, and one copy of the data should stay permanently out of the agent's reach.

The author reports his own setup works this way: a hook reads each command before it runs and recently blocked a git checkout -- . that would have wiped uncommitted changes, forcing the agent onto a safer path using git stash. He acknowledges the guardrail has limits — a misresolved variable or a Python script can still get through — which is why logs and backups remain essential.

Why it matters

The dangerous case is not an AI that fails often; it is an AI that is almost always right. High reliability erodes the attention of whoever is supposed to catch the rare failure, whether that person is holding a steering wheel at midnight or clicking through permission dialogs on the fiftieth prompt of a session. The road data, the NHTSA investigation and Anthropic's approval numbers all measure the same decay. Any safety case that rests on "a human is watching" is therefore building on a documented failure mode, and the practical response is to engineer the control into the system itself: hard blocks on destructive actions, logs, and recoverable state, designed as if nobody were watching.