DishwasherBench scores AI models on loading a physics-simulated Bosch dishwasher

What the benchmark does

Composo has released DishwasherBench, an open-source project that turns loading a dishwasher into a measurable task for AI systems. Surfaced on Hacker News' front page, the repository describes a browser game called Spotless, built around a 3D reconstruction of a single Bosch SMV4HAX40G — modelled from photographs and tape readings of the real appliance, including both baskets, tine rows, spray arms and the Eco 50 wash programme.

Players drop items into the machine under deterministic rigid-body physics, using the Jolt engine. The wash is then simulated — water jets, droplets, run-off, soil and detergent chemistry — and points are awarded for every item that comes out clean. From the second of seven levels onward, the available items no longer all fit, so loading becomes a spatial planning problem with real trade-offs.

One game, three ways to play

According to the repository, the same game can be played three ways: in a browser by people, through an MCP server by AI agents, and via a harness that runs scripted players and models and writes episode records that anyone can replay and wash again to verify. The browser interface includes a "suggest spots" helper that numbers the free seats and shows the exact pose data a program would send, underlining that humans and models face identical mechanics.

Early results are thin. Two models have played through an OpenAI-compatible endpoint, producing six episodes across three levels at the official scoring tier. No human baseline exists yet, and an Anthropic API path has not been run against the live service. Each level ships with reference loads from three scripted players — careful, careless and random — and six of the seven levels pass the benchmark's gates on those loads; the tutorial does not.

Compute demands are modest. A cold official wash of the largest level takes 4.9 minutes on the reference MacBook Pro (Apple M3 Pro, 18 GB), against a 30-minute budget. A faster preview tier agrees with the official tier on 89% of verdicts for a standard 13-place load and 97% on the benchmark levels.

How the simulation was checked

The validation documentation is unusually candid. The simulation is calibrated against exactly one test wash: four egg-yolk plates run on Eco 50. A prediction written beforehand got the ordering of the plates and the form of the residue right, but was ten to thirty times too optimistic about how much soil remained on two oven-dried plates. Calibration then adjusted a single parameter — the share of the long soak during which the pump runs — from 0.5 to 0.8. The project notes that the record showing the prediction came first sits in an unpublished development history, so that ordering cannot be independently verified.

Against published class means for a sister machine, the Bosch SMV6ECX51E, the simulation lands within 0.1 of a score point for cutlery, 0.2 low for crockery and 0.1 to 0.5 low for glass — though the documentation is explicit that these are fitted values, not predictions. Of 48 loading phenomena drawn from the literature and encoded as executable checks, 17 pass, 21 fail in size or sign, one scenario is faulty and nine require mechanisms that have not been built.

Known blind spots

The project states plainly what a score currently reflects: whether a load was packed well and whether cups, glasses and pots went in mouth-down. It does not yet capture whether plates and cutlery were arranged so that they wash clean. Fallen or leaning plates come out clean, which the authors describe as most probably wrong; fat-stained items clean even without detergent; and verdicts for tea, coffee, wine and milk dried on glass are unreliable. Drying, spots and film on glass, items shifted by the jets, and soil settling onto clean pieces are not modelled at all, and only the Eco 50 programme is calibrated.

This is a version 0.1.0 release, tested on a single macOS machine. It needs Node 20.19 or newer, pnpm 9, a GPU that Dawn can open for WebGPU, and Chrome, with roughly 680 MB for the clone and one to two GB of memory per official wash. Linux and Windows are untested.

Why it matters

Most model benchmarks test text or code; DishwasherBench probes spatial planning in a physical environment with genuine consequences, where a blocked detergent tray or an upside-down mug changes the outcome. Because humans and agents play the identical game with verifiable episode records, it offers a route to meaningful human-versus-model comparison in embodied reasoning without requiring a robot. The transparent publication of weaknesses — a single calibration wash, 21 of 48 literature checks failing — sets a high bar for honesty in simulation benchmarks, though it also means any model rankings it produces should be read as provisional.