Kev ships self-trainable Qwen3.5 decision models for local agent calls

Small models for structured decisions

Kev, a project published on GitHub under the handle jaredpalmer and surfaced on Hacker News' front page, is a family of compact decision models built on Qwen3.5 base models. Instead of generating prose, Kev answers structured questions about input text: yes/no judgements, multiple-choice picks and ratings. According to the repository, the design follows the architecture described in "Jev's Architecture Unmasked," and the project ships both pretrained weights and the training code needed to build your own variants.

The current generation comes in three sizes — 0.8B, 4B and 9B parameters — all fine-tuned from Qwen3.5-Base checkpoints with the same training data and settings. The recommendation in the README is to start with Kev-4B, step up to Kev-9B when accuracy and calibration matter more than memory, and use Kev-0.8B when footprint is the priority.

Questions, probabilities and the System One API

The server exposes a single endpoint, POST /v1/systemone, which takes a piece of state text plus a set of questions. Three question types are supported: "noul" returns a probability of yes; "choice" picks from 1 to 255 named options, each with an optional description; and "score" maps the input onto 2 to 255 ordered levels and returns a mean level index with a legend. Several questions can travel in one request — they share the input text but cannot read each other's answers, a property the project treats as important enough that its playground includes presets for testing question isolation.

Responses include probability distributions and confidence values rather than bare labels. In the README's worked example, a support ticket about late, wrongly sized shoes and a duplicate charge, the routing question splits its probabilities across returns (0.47), shipping (0.28) and billing (0.25) — which is the point of getting a distribution back when a ticket touches several departments. The project notes that its confidence figures approximate a TypeSafe formula that is not public, and are not measured accuracy rates.

The API deliberately matches TypeSafe's System One, so TypeSafe's Python SDK can be pointed at a local Kev server with no other changes.

How the benchmarks look

Each model is reported on development and test splits, split further into "trained sources" (held-out examples from training datasets) and "new sources" (datasets and rule types the models never saw). On new sources, Kev-9B scores 0.812 development / 0.837 test accuracy with Brier scores of 0.291 / 0.243; Kev-4B reaches 0.794 / 0.832; Kev-0.8B manages 0.643 / 0.668.

The comparison against the hosted Jev model is less favourable: Jev posts 0.857 accuracy and a 0.211 Brier score on the new-source development set, leaving Kev-9B roughly 4.5 points behind. The README is explicit that this is not a controlled comparison, since Jev's training data is unknown.

A cleaner experiment exists between generations. The earlier Kev family used Qwen3 bases with identical data and settings, so only the base model changed. On the test set, Kev-9B leads its Qwen3-8B predecessor by 7.3 points (95% CI +2.8 to +11.7) with a Brier score 0.08 lower, Kev-4B is 2.9 points ahead (−0.9 to +6.4), and Kev-0.8B is 4.8 points ahead (+0.2 to +9.3). The Qwen3 weights remain published and are the faster option on a Mac, but are no longer developed.

Running and training it yourself

Setup requires Python 3.12+ and uv; the quick start serves Kev-4B locally, downloading the adapter and base model on first run. CUDA and Apple Silicon are both supported, and the README says the 4B and 9B models fit a 32 GB Mac in bf16. The sample response reports 495 ms latency for Kev-4B in bf16 on an Apple M5. A Node-based playground lets users compare asking questions packed together versus separately, run a choice question under six option orderings, probe question isolation and fake delimiter tokens, and play chess against the model, where legal moves are choice options and a score question rates the position.

Weights are distributed through a model collection and a GitHub release with tarballs and SHA-256 checksums, and the repository includes training code and evaluation data. On the input side, delimiter-like strings in user text are escaped before tokenization and invalid requests return a 422.

Why it matters

Agent pipelines routinely spend a large generative model on calls that are really classification: routing tickets, flagging escalations, scoring urgency. Kev's pitch is that those decisions can be handled by a small, locally hosted, fine-tunable model that returns calibrated probability distributions — cheaper to run, easier to audit, and keepable on private infrastructure. The benchmarking is unusually candid about which comparisons are controlled and which are not. The roughly 4.5-point gap to hosted Jev shows self-hosted open models are not yet at parity, but the Qwen3-to-Qwen3.5 result demonstrates that base-model improvements flow through to downstream decision quality. For teams already building against the System One API, Kev works as a drop-in local backend.