Strot compiles LLM intent once into a local decision model that runs every record at zero marginal cost

An open-source project named Strot surfaced on Hacker News's front page with an unusual pitch: treat the LLM as a compiler, not a service. You state your intent once in plain language, an LLM turns it into a frozen, self-contained Python artifact, and from then on every classification, scoring or routing decision runs locally — with no model calls, bounded output and explicit confidence scores.

What the compiler actually does

According to the project's GitHub documentation, the pipeline has two distinct phases.

At compile time, an LLM — a Claude model in the documented example — is invoked exactly once per intent and schema. It synthesizes labels, projections and handlers, which must first pass a "golden gate": a check against answers you already know. Once the artifact clears that, a sample is annotated to fit a confidence threshold, and the result is written out as a plain Python file.

At runtime, no LLM is involved. A small decision model, by default the 144M-parameter Julia-1 from SupersonicLabs running on CPU, arbitrates each record in roughly 60 milliseconds, according to the project. Every output arrives as typed JSON carrying probabilities and a confidence score, and records the system is unsure about abstain instead of guessing.

Two programming surfaces

Strot exposes a pair of APIs. run() routes each record to one action in a developer-supplied registry — refund requests versus crash reports, for instance — and executes the compiled handler. decide() answers typed questions per record: booleans, bounded scores and multiple-choice options, each with natural-language instructions, returning probabilities and confidence alongside the answers.

The benchmark claims

The project publishes its own measurements, which it says were produced with the real Julia-1 model on real datasets inside clean Docker containers. On a 398-record grounded routing task, the compiled artifact scored 92.5% accuracy against 74.8% for the same model used naively. On AG News with labels the system had not seen, it reported 86.7% versus 72.7%. A typed sentiment comparison against a leading hosted decision API's own example came out near-identical, 1.79 versus 1.8. Per record, the project claims about 60 ms of latency, zero marginal cost and zero LLM calls. The repository includes the benchmark code and, notably, the failures, though these figures have not been independently verified.

Guardrails and guarantees

Several design choices are meant to make the approach production-safe:

• Artifacts are cached by content, and schema drift triggers exactly one recompile rather than silent misbehaviour.
• An artifact must pass the developer's known answers before it is allowed to serve traffic.
• Calibration uses temperature scaling plus split conformal prediction, giving a marginal statistical bound: dispatched decisions are wrong at most a chosen fraction of the time, and everything else abstains.
• Because runtime output is selection and computation over your own registry rather than generated text, the project claims zero structural hallucination.
• The decision model is a swappable slot, accepting the local default, hosted APIs or any drop-in specialist.

Availability

Strot is MIT-licensed, ships on PyPI as strot-ai with optional extras for the local model, hosted runtimes and the Anthropic synthesizer, and can prefetch its local checkpoint for fully offline deployment. The default decision model is Apache-2.0 licensed. The README warns that a similarly named julia package on PyPI is unrelated. The project says it implements ideas from an academic paper, STROT (arXiv:2505.01636).

Why it matters

Per-call pricing is the default economics of LLM decision-making today, which makes high-volume tasks like support triage, moderation and lead routing awkward to cost. Strot inverts that: pay for one synthesis pass, then decide on unlimited records on hardware you control, offline and with predictable latency. Equally important is the abstention design — uncertainty becomes an explicit, statistically bounded signal rather than a silent wrong answer. If the reported accuracy holds up under outside evaluation, compile-once decision artifacts could become a serious alternative to hosted decision APIs for bulk workloads.