AI agent CAD benchmark: OpenSCAD and CadQuery both deliver, but they fail differently

ModelRift, a platform that generates every model on its site as OpenSCAD code, has published a controlled benchmark of that language against its most credible code-first rival, CadQuery, a Python library built on the OpenCascade B-rep kernel. The question, according to the write-up, was deliberately narrow: which tool can an AI agent drive to a correct, printable, functional part with nobody watching? Ease of hand-writing was out of scope. All six resulting parts came out printable and geometrically clean, and capability was not where the two tools separated. The character of their failures was.

How the test was run

Six agents ran, one per tool-and-task cell, each an isolated general-purpose subagent using Claude Opus 5 with the 1M-token context window, driven through Claude Code with no cross-talk between cells. With a single agent per cell, ModelRift cautions, some of the spread reflects agent variance rather than tool difference. Each agent was capped at 12 versions, told never to fake success, and required to quote every failure's error text verbatim. The runs were unattended, on CadQuery 2.8.0 with Python 3.14 and OpenSCAD 2026.06.12, both on an M-series Mac.

To level the field, ModelRift ported its in-house OpenSCAD agent skill to CadQuery operation by operation. Since CadQuery has no CLI renderer, the port needed a small offscreen renderer plus B-rep validity metrics in place of OpenSCAD's CSG status line; both skills carried identical printing design rules for wall thickness, clearances and overhangs. One asymmetry survived: the CadQuery skill includes an API cheatsheet the OpenSCAD side does not need, because the model already knows OpenSCAD syntax, which helps with syntax and not geometry.

Nothing was taken on trust. Every final STL passed through an independent parser that reads the file directly and reports triangle count, volume, watertightness, non-manifold and boundary edges, flipped faces and connected components. All six verdicts came back clean, a verification step the authors say proved more important than they anticipated.

The tasks and the totals

The three tasks escalated in difficulty: a wall-mounted shelf bracket with gussets, countersunk holes and fillets; a two-part snap-fit enclosure for a 50 x 26 mm PCB whose lid and body had to genuinely fit; and an M24x2 threaded hose-barb adapter demanding a true helical thread, with stacked rings explicitly banned.

Summed across tasks, the totals are strikingly even. Both tools needed exactly 11 versions. OpenSCAD accounted for 473 lines of code, 297k tokens and roughly 34 minutes of agent wall-clock; CadQuery for 573 lines, 347k tokens and about 40 minutes. CadQuery raised five tool-level errors to OpenSCAD's two, while OpenSCAD produced five instances of silent wrong geometry to CadQuery's four. Geometry recomputation split hardest: 12 to 43 ms per recompute for OpenSCAD against roughly 1.6 to 2.0 seconds for CadQuery.

Where the tools diverge

On the bracket, the dividing line was corner rounding. OpenSCAD rounds a 2D profile and extrudes it, indifferent to how the solid was assembled; CadQuery must first select target edges with box selectors, and edge selection is where the difficulty sits. The CadQuery agent spent about a third of its run on one failure, a fillet throwing 'BRep_API: command not done', an error naming neither the edge nor the radius, then bisected by hand to find the cause: two R3 fillets do not fit in a 4 mm wall. OpenSCAD compiled correct geometry on the first attempt and finished in two versions.

The enclosure favoured CadQuery's derived parameter chain, where changing the lip depth moves rim, plate, slots and groove together. More consequential for agents was verification style. OpenSCAD's echo prints numbers that a person must read; CadQuery's assert fails the build the moment box and lid interfere. For unattended generation, the assert closes a feedback loop the echo leaves open. OpenSCAD needed eight versions to CadQuery's five, three of them spent on boolean hygiene, cleaning slivers thrown off by tangent and coincident faces.

The thread task produced the upset. OpenSCAD finished in a single version, correct on the first compile, with a 43 ms recompute and no library. Lacking a sweep operation, the agent wrote the helix as raw vertex and face arithmetic, a four-point ISO profile at 96 sections per turn emitted as one polyhedron. It worked, but nothing in the toolchain validates such code: an inverted winding order would go unreported. CadQuery's thread run, per the results table, took three versions with one raised error and one silent geometry fault.

Why it matters

For anyone building AI-driven, code-first CAD tooling, the benchmark's value lies less in a winner than in its failure taxonomy. Capability is table stakes here; both kernels reached correct, printable parts. What separates the tools is how they fail: opaque kernel errors that force costly bisection on one side, and silent wrong geometry, the most dangerous class when nobody is watching, on both. The mitigations are methodological: asserts that convert bad geometry into a raised error, cheap recomputation that tightens the inspect-and-fix loop, and independent verification of the output file rather than the generating tool's self-report. One caveat worth holding: ModelRift ships OpenSCAD as its platform default and was re-testing its own choice. Even so, the template of matched agent skills, an untrusted parser and verbatim error capture is reusable by anyone evaluating geometry kernels for agentic pipelines.