Cactus releases Needle 3, an 8-29MB model built for on-device tool calls

What Cactus announced

Cactus has released Needle 3, a foundation model that ships as a single 8-29MB binary and is designed to run natively on phones, wearables, robots, smart home devices, cars and microcontrollers. The announcement was published on the company's site and surfaced on Hacker News as a Show HN post.

The company says it deliberately traded general chat ability for automation performance: the model is built for tool calls, structured extraction and text embeddings rather than open-ended conversation. On those narrower jobs, Cactus claims Needle 3 beats systems ten times its size on mobile tool calling and matches models two to three times larger on extraction — and that a fine-tuned 4-layer subnetwork can match DeepSeek V4 Flash on downstream tasks.

One set of weights, twenty sizes

The core design idea is what Cactus calls intelligence laddering. Every layer of the model is a sub-network with monotonically increasing capacity, so developers can use the same weights as anything from a 2-layer to a 20-layer model, and each rung can be fine-tuned independently. A wearable app could ship a small tuned subnetwork while a phone ships the full stack.

Under the hood, Needle 3 is a 29-121M parameter Laddered Simple Attention Network rather than a conventional transformer, quantised to a 2-bit format Cactus calls CQ2. Training used 360B tokens of proprietary structured data. Cactus reports decode speeds of 400-4,000 tokens per second and prefill of 1,000-10,000 tokens per second on a Raspberry Pi 5, and claims the architecture uses less than half the MFLOPs per token of a transformer of the same configuration.

Three jobs: calls, extraction, embeddings

For tool calling, the model reads the functions an app exposes, picks the right ones and fills in arguments from what the user said. According to Cactus, multi-intent requests produce multiple calls in order, and a request no tool covers returns an empty list instead of a guessed call.

For extraction, a developer declares a schema — a Pydantic model in the Python package — and hands over messy text such as an invoice, a booking or a notification, getting typed fields back. A decode grammar guarantees the output parses, and Cactus says the capability transferred well to classification tasks too.

The same weights also produce sentence embeddings, enabling on-device semantic search, routing a query to the closest of many tools, and merging near-duplicate alerts without any data leaving the hardware.

Guardrails and fine-tuning

The Python package leans on signatures and docstrings as tool definitions: decorate a function, call run(), and the engine picks the call, executes it, feeds the result back and returns the final response. Regular-expression triggers can force a request toward a named tool when a description cannot enumerate every phrasing, restricting decoding to the matched tools and requiring a call.

Every response carries a confidence score from a calibrated head. Below a floor of 0.1, calls are withheld into a suppressed_calls field rather than executed; above it, developers can route on the score — act immediately when it is high, ask for confirmation when it is middling. Cactus also advises narrow, plainly described tools, since every extra tool in a turn increases the chance of misrouting.

For customisation, developers can run LoRA fine-tuning on the frozen base at all 20 layers and export a 4-bit file for any subnetwork. The paid Cactus Platform adds dataset management, evaluation tracking and full fine-tunes on the company's infrastructure. Each deployment target ships a prebuilt engine under 1MB that loads the needle3.cact weights at start; the inference engine is fetched from Hugging Face and cached on first install.

Read the benchmarks with care

The performance claims are Cactus's own, and the announced methodology mixes serving setups: Needle subnetworks were scored through the shipped 2-bit binaries, baselines ran at f16 under vLLM, and DeepSeek V4 Flash went through its cloud API. Cactus reports that fine-tuning on the DroidCall dataset lifted every subnetwork by 18 to 36 points, with tuned models from 4 layers — about 29M parameters — upward passing DeepSeek V4 Flash. None of these figures have been independently verified yet, so treat the headline comparisons as vendor benchmarks until third-party evaluations appear.

Why it matters

Needle 3 sits squarely in the push to run useful AI on the device rather than in the cloud, which buys offline operation, lower latency, no per-request API cost and data that never leaves the hardware. The laddering concept is the interesting engineering bet: one training run yields a family of models spanning a microcontroller to a phone, and a product line can climb the ladder as accuracy demands grow. The release also reflects a broader shift in how small models are judged — not on general chat skill, but on whether a task-tuned specialist can match a much larger generalist on a narrow job. Independent evaluation will decide whether the numbers hold, but the shape of the product — tiny, quantised, embedding-capable, tool-first — marks where mobile and embedded AI is clearly heading.