Open-source decision-cascade escalates to a big model only when the math says it pays off

An open-source project called decision-cascade targets a tradeoff every AI agent builder eventually hits: large models are accurate but expensive on every request, while small local models are cheap but fail on the hard cases. According to a post on dev.to by the project's author, the package answers with a tiny local model first and escalates to a larger external service only when the expected benefit justifies the price of the call. It is installable from PyPI.

Why a confidence threshold fails

The author's first version used the obvious rule: if the local model's confidence dropped below 0.6, escalate. It failed in practice, and the reason is worth internalising. Confidence and benefit do not line up. You can be deeply unsure about a question where being wrong costs nothing, and fairly confident about one where an error is expensive. A pure threshold ignores both the cost of a mistake and the price of the external call, so the cascade escalates on harmless cases and skips dangerous ones — the opposite of what you want.

Escalation as an expected-value decision

The current design compares two numbers per request. The first is the expected utility of answering locally: the local model's confidence, penalised by the probability of it being wrong multiplied by what an error costs in that situation. The second is the expected utility of escalating: the external model's estimated accuracy minus the price of the call. If the second number is larger, the request escalates; otherwise it stays local.

One detail matters: the external model's accuracy is not a fixed constant. It is updated online through a feedback endpoint as users report whether the external answer was right, so the cascade adapts to the actual quality of whatever backend is attached. The post names Clef, Laya and Perplexity as plug-in options, alongside a custom endpoint.

A hard gate for small data

Adding a learned risk model initially made things worse: trained on eight examples, it overfit badly enough to underperform the plain threshold it was meant to replace. The fix is a hard gate. If the learned model does not beat a simple confidence threshold on held-out data, the system automatically falls back to the threshold, and the API response exposes the reason as risk_gate so operators can see when it happens. The author presents this unglamorous safeguard as what separates a production component from a research demo.

Reported benchmarks

The author reports benchmarks run on a 16GB GPU with 200 evaluation examples, comparing the 124M-parameter cascade against Clef-Flash (9.4B parameters, GGUF Q4_K_M) and Laya (421M parameters). On AG News classification, decision-cascade scored 0.873 accuracy at 10ms p50 latency, versus 0.855 at 130ms for Clef-Flash and 0.662 at 16ms for Laya. On RouterBench it scored 0.542 at 21ms, against 0.442 at 170ms for the larger model.

The author is careful about scope: on decision tasks such as classification and escalation choices, a small model with task-specific heads and calibrated escalation can beat a much larger general model. On free-form reasoning and generation, the larger model still wins easily. Evaluation scripts and raw results are said to be reproducible from the repository.

Two games serve as stress tests where mistakes carry a visible cost. In chess against Stockfish, escalating on 19% of moves closed half the gap to always searching deeply, at 38% of the full-search cost. In a Pac-Man-style maze, a cautious policy reached 94% of the maximum score with 33% escalations, which the author says is twice the greedy strategy.

Stated limitations and motivations

The post is blunt about weaknesses. The default model heads are random until you run supervised fine-tuning or load a pre-trained weight pack. The risk model needs examples before it adds anything over a threshold, and the hard gate admits this in the response. Public benchmarks are not your support tickets, so domain calibration falls to the operator. Cold-start inference on CPU takes a few seconds after a model load, with a hash encoder offered as the fastest way to smoke-test the pipeline.

Two motivations drive the local-first design. Cost: by the author's estimate, 70–90% of decisions in a typical product are routine, so escalating all of them spends money without a quality gain. Privacy: some decisions should not leave the building, and a local cascade keeps sensitive cases on-prem while still escalating ambiguous ones. The project speaks the SystemOne protocol and can run as a gateway to external services or fully self-contained.

Why it matters

Every team wiring agents into a product faces the small-versus-large model question, and most resolve it with a static confidence threshold or an all-or-nothing commitment to one model. Decision-cascade's useful idea is treating escalation as an economic decision that weighs the cost of an error against the cost of a call, rather than how confident the local model happens to feel. The hard gate and the candid limitations section are equally instructive, offering a pattern for shipping a learned component you cannot yet fully trust.

The numbers deserve scrutiny: they come from a single self-published post and the wins are confined to classification-style decision tasks, not generation. But the package is pip-installable and the evaluation is claimed to be reproducible, which makes the pattern cheap to test against your own traffic — arguably the only benchmark that matters here.