Measured benchmarks put 8B LLM inference at 10–35 tok/s on CPU, NPU and iGPU hardware

Measured numbers for GPU-free inference

A write-up on dev.to has published hand-measured token rates for running large language models on CPUs, neural processing units and integrated GPUs, and the headline finding is that single-user local AI works on ordinary hardware. The author's opening example: a consultant's three-year-old office laptop with 16 GB of RAM and no discrete GPU ran a quantized 7B model at roughly 11 tokens per second, answering questions over the firm's own contract PDFs. Not a production chat service, but a private assistant with zero per-query cost and no data leaving the machine.

The technical claim underpinning the numbers is that single-user token generation is memory-bandwidth bound rather than compute bound. Generating a token requires one forward pass, and at the small batch sizes one user produces, what matters is how fast weights stream from RAM, not raw FLOPS. That fact explains most of the results.

What the hardware actually delivers

According to the post, the measured speeds on current consumer machines break down as follows:

• 2021 office laptop, 16 GB RAM: Llama 3.1 8B at Q4_K_M quantization (~5 GB), roughly 10–12 tokens per second
• Apple M2/M3 MacBook Air: same model at 25–35 tok/s, credited to fast unified memory shared between CPU and GPU
• Apple M2 Pro/Max: a 14B model at Q4_K_M (~9 GB) at 20–30 tok/s
• Recent laptop CPUs with NPUs (Intel, AMD, Qualcomm): 7–8B models with NPU offload at 15–30 tok/s, at lower power draw
• DDR5 desktop CPU with 16 cores and no GPU: 8B at Q4_K_M at 15–20 tok/s; a 14B model on 32 GB RAM manages 8–12 tok/s
• iGPU offload with shared memory: only a modest gain over CPU alone

For context, the author puts comfortable reading speed at about 20 tok/s and says interactive chat starts feeling sluggish below 8 tok/s. Overnight batch jobs, such as summarising hundreds of documents, don't care about interactivity at all, which changes the calculus entirely.

Prompt length dominates the experience

The post draws a distinction between the two phases of generation. Prompt processing, or prefill, is compute-bound and slow on CPU: a 2,000-token prompt can take a second or two before the first output token appears. Token generation, or decode, is memory-bandwidth bound and steady once the model is warm. The practical consequence is that short prompts with long answers feel fine, while long prompts with short answers feel sluggish — pure prefill time. Capping context and keeping inputs tight is framed as a latency lever, not a quality compromise.

The four-part stack

Per the author, every serious CPU inference setup rests on the same components:

• Quantized GGUF models: shrinking 16-bit weights to 4-bit integers cuts memory and bandwidth roughly fourfold. Q4_K_M is the recommended default at ~5 GB for an 8B model; Q8_0 offers better quality at ~9 GB; Q3_K_M fits in 4 GB but with noticeable quality loss.
• llama.cpp: the C/C++ reference runtime, written to be fast on CPU and able to offload layers to a GPU or NPU on hybrid hardware via its layer-count flag.
• A server layer: llama.cpp ships a server binary that speaks the OpenAI-compatible chat completions API.
• Ollama or llama-cpp-python as glue: a single model pull plus ollama serve exposes a local OpenAI-compatible endpoint, so any existing client works by changing only its base_url.

The recommended starting point is a 7–8B model at Q4_K_M — Llama 3.1, Qwen 2.5 or Gemma — on machines with 16 GB of RAM, stepping down to a 3–4B model on 8 GB or older hardware.

Why it matters

These benchmarks give developers a realistic decision map instead of marketing claims. Private per-person assistants and overnight batch workloads don't need a GPU at all; multi-user production serving does. The numbers also reframe hardware conversations around memory bandwidth and prefill latency rather than FLOPS, which explains why a fanless MacBook Air beats a bigger Windows laptop for this workload. And because local runtimes expose OpenAI-compatible APIs, existing applications can move to on-device models with a one-line configuration change — turning "we need expensive GPUs for AI" into a question of what you actually run, and for how many users.