Rust and Candle run Gemma 4, Qwen 3 and Phi-4-mini on CPU with zero Python

A dev.to post by Hossein Mortazavi walks through building a pure-Rust command-line engine that runs three current small language models — Google's Gemma 4 E4B, Alibaba's Qwen 3 4B and Microsoft's Phi-4-mini 3.8B — entirely on an ordinary CPU, with no Python runtime, no CUDA toolchain and no GPU. The project extends an earlier experiment in which the same author ran Microsoft's Phi-3 on consumer hardware using Hugging Face's Candle framework, and argues that compiled-language CPU inference has matured from a curiosity into a workable production architecture.

A Python-free inference stack

The engine is built on Candle 0.8, Hugging Face's machine-learning framework for Rust, together with the hf-hub crate for fetching weights, the tokenizers crate for encoding prompts and clap for the command-line interface. According to the post, the motivation is deployment: for edge devices, IoT gateways or cheap CPU-only cloud instances, the overhead of a Python runtime matters. A Rust program compiles to a single executable, which the author contrasts with the gigabyte-scale Docker images and elaborate accelerator setups that typically accompany local AI tooling.

All computation runs against Candle's explicit CPU device, and models load from quantized GGUF files — the same format used by llama.cpp-style runtimes — keeping weights small enough for machines without dedicated video memory.

Three models, one binary

Instead of tying the tool to a single architecture, the code defines a model-family selector that maps each option to its Hugging Face repository, GGUF file, tokenizer and chat template. Qwen 3 uses the ChatML format, Phi-4-mini uses its own end-token convention, and Gemma 4 uses Google's turn markers. Phi-4-mini and Gemma 4 weights come from quantized conversions published by bartowski, while Qwen loads from the official Qwen repository; all three run at 4-bit Q4_K_M quantization.

Generation follows the familiar two-step pattern: a prefill pass processes the prompt, then tokens stream one at a time with temperature sampling (0.6 by default) and a fixed random seed. The tool also times the loading step, printing the cold-start duration as weights come off disk.

The post characterizes the 3B-to-4B contenders this way: Gemma 4 E4B is an edge-first architecture with strong multilingual and instruction-following behaviour, Qwen 3 4B is particularly capable at coding, mathematics and structured JSON output, and Phi-4-mini — the successor to Phi-3-mini — was trained on synthetic textbook data for logical reasoning.

What gets measured

The benchmark suite targets the three numbers that decide whether CPU inference is practical: RAM usage, cold-start latency for loading model weights, and token generation speed. The author describes these as real-world deployment metrics, since a model that fits in memory but loads slowly or generates sluggishly is a poor fit for interactive or service workloads. As with any single-machine benchmark, absolute figures will depend on the specific CPU and memory available.

Why it matters

Most local-AI tutorials assume a Python environment and, sooner or later, a GPU. A compiled Rust binary that runs 3B-to-4B parameter models on hardware people already own changes the economics of local inference: no container image measured in gigabytes, no accelerator drivers to install, and memory use predictable enough for consumer machines.

It also widens where AI can run at all. Edge gateways, embedded controllers, CI pipelines and low-cost virtual machines are typically CPU-only, and a language with a small runtime footprint and strong memory-safety guarantees suits long-running, unattended inference services. According to the dev.to post, the newest small models are designed with exactly these constraints in mind, which means the remaining bottleneck is increasingly the deployment stack rather than the models themselves.