CauterRule v0.3.0 lands on PyPI with domain-scoped replay that more than doubled recall

What shipped

CauterRule v0.3.0 is now available on GitHub and PyPI. According to the release post on dev.to, the package is an open-source sidecar for AI agents that mines recurring failures from execution trajectories, proposes standing rules that would have prevented them, and verifies each candidate by replaying it against reference runs before promotion. The release bundles a full CLI, framework adapters, a rule lifecycle, and a pack ecosystem with official rule packs.

The version's headline change, however, is not a feature but an evaluation fix.

The metric that hid good rules

Across two consecutive field tests, one figure refused to budge: golden recall of 0.087. Both reports pointed at matcher calibration as the highest-value engineering target. The v0.3.0 work concluded the matcher was not the problem — the reference pool was.

Recall was computed as the failures a candidate rule prevents, divided by all failures in the corpus, roughly 200 of them. A genuinely useful rule that stopped three real git failures scored 3 out of 200, or 0.015. No admissions threshold can accept a score that low without also accepting noise, so good rules were being rejected across the board. The underlying assumption, as the dev.to post puts it, was that every failure in the pool is a fair test case for every candidate — which is false whenever rules are domain-specific, and for a real rule engine they always are.

Scoping replay to the domain

v0.3.0 filters the reference set to trajectories sharing the candidate's source domain before scoring. The overall pool actually grew, from 230 to 444 references, but each candidate is now judged against a small in-domain slice: 19 references for git, 30 each for python and docker. The same three-failure git rule that scored 0.015 now scores roughly 0.11.

One more change shipped alongside: the pass threshold fell from 0.8 to 0.5. The author flags this openly as a recalibration to an honest scale rather than a fix, and concedes that landing both changes together blurs attribution — the recall gains belong to scoping, while the pass-rate movement reflects both.

The numbers

The v0.3.0 field test evaluated two cloud models, gpt-4o-mini and llama-3.1-8b, across 40 corpora and 4,768 trajectory runs. With no change to the models, the prompts, or the core matcher scoring, golden recall for gpt-4o-mini moved from 0.068 to 0.170, a 2.5x gain, and for llama-3.1-8b from 0.104 to 0.228, a 2.2x gain. On the failures-positive measure, the two models went from 0.068 to 0.182 and from 0.104 to 0.277, both 2.7x. Golden pass rates reached 40 percent and 50 percent respectively. That the improvement held across both models is, per the post, the signature of an evaluation bug rather than a capability ceiling.

What remains broken

The fix is candidly described as partial. Recall of 0.170 to 0.228 sits far below the roughly 0.70 ambition, and doubling a small number still leaves a small number. The matcher's paraphrase gap is untouched: with semantic matching running at a 0.2 blend weight, it cannot connect "non-fast-forward" to "Updates were rejected because the remote contains work that you do not have locally" — the same event expressed in different tokens. Scoping may also flatter domains rich in references while penalizing thin ones, which has not been measured. Open questions include whether a higher semantic weight would help, whether failure-class rather than domain is the right scope, at what reference count scoping stops helping, and how to handle unlabeled candidates without reintroducing the original bug.

Why it matters

There are two reasons this release deserves attention. First, the tool itself: converting one-off agent failures into durable, replay-tested rules is a pragmatic route to agent reliability that requires no retraining or prompt surgery, and it is now a pip install away.

Second, the lesson generalizes well beyond this project. When every model scores uniformly badly on a metric, the dev.to post argues, suspect the denominator before the model. Evaluation-set composition is a product decision, not a detail — "recall against all failures" sounds rigorous but answers a different question than the one you actually care about, and recall should always be read as a floor that shrinks with every reference added.