NEON kernel patch lifts llama.cpp prompt processing up to 6.5x on Arm CPUs

A missing kernel, now written

llama.cpp has gained a native Arm kernel for the matrix-multiply routine it uses to process prompts, lifting throughput for low-bit quantized models by as much as 6.5x on Arm server cores. The work is published as a GitHub repository, silicon-forge-arm-neon-study, which surfaced on the Hacker News front page and contains the kernel, a patch against llama.cpp v0.9.0, benchmark data and tooling.

According to the repository, llama.cpp processes prompts several times slower on Arm CPUs than on x86 for low-bit quantized models, and the cause is a single routine. The tiled K-quant matmul, added in September 2026 through llama.cpp PR #27851 by jbooth with bartowski, shipped inner kernels only for x86: AVX-512 VNNI, AVX2 and plain AVX. On Arm, the formats without a kernel of their own — Q3_K, Q2_K and the IQ family — fell back to a generic path that unpacks the weight data again for every dot product.

What the patch does

The patch adds a NEON kernel built around the SDOT dot-product instruction and enables it when the compiler targets an Arm CPU with the dot-product extension. It is deliberately small: 164 added lines across two files, with the routine's portable driver, tile format, unpackers and tests left untouched.

Two design points carry the speedup, per the repository's write-up. For every supported format, the weight code minus its bias fits in a signed byte, so the kernel subtracts the bias while loading and a single signed dot product produces the bias-corrected sum in exact integer arithmetic. Separately, the 16 rows of an activation band are interleaved in place — the same layout the x86 VNNI kernel uses — so one SDOT-by-lane instruction accumulates partial sums for four different rows, with nothing reduced afterwards.

Measured gains

Benchmarks used Qwen2.5 1.5B-Instruct with 4 threads and a 512-token prompt on three Arm server generations, rented as AWS Graviton2, Graviton3 and Graviton4 (Neoverse N1, V1 and V2):

• IQ3_M: 11.5 to 74.5 tokens per second (6.5x) on N1; 16.6 to 80.8 (4.9x) on V1; 21.7 to 92.6 (4.3x) on V2
• IQ3_XS: 12.6 to 74.9 (5.9x) on N1
• IQ2_M: 13.9 to 60.6 (4.4x) on N1
• Q3_K_M: 31.4 to 64.5 (2.1x) on N1
• Q2_K: 25.7 to 50.6 (2.0x) on N1

Formats that already had an Arm kernel through an older repack path — Q4_K_M, Q5_K_M, Q6_K and Q8_0 — show no change, and generation speed is unchanged because generation does not use the routine. Before and after figures come from the same binary: the environment variable GGML_CPU_TILED_MM=0 switches the routine off, reproducing stock v0.9.0 behaviour on Arm.

Verification and limits

The kernel passes llama.cpp's own test for the routine — 15 of 15 cases, plus 88 of 88 in its randomised fuzz mode — on all three chips, and the repository documents a line-by-line reading review that found no correctness bug. Missing, by the author's own account: review by a human kernel engineer, sanitizer builds and comparison of model output on real prompts.

The stated limits are extensive. Only rented Arm servers were measured, not phones, Apple Silicon or Windows on Arm, though the instruction used exists on phone cores from Cortex-A55 and A75 onward. One small model was tested, at 4 and 8 threads, and the reviewer expects gains to shrink, and possibly reverse, above roughly 16 threads. Only prompt processing benefits, and Arm throughput, while much improved, still sits below the x86 reference — what bounds it now has not been profiled. Only 128-bit NEON is used; SVE and I8MM are not. The related project ik_llama.cpp already ships its own NEON kernels for these formats, which were not benchmarked here.

Deliberately not upstream

The repository states the patch has not been submitted to llama.cpp: the project's contribution rules allow AI-generated code only when a human contributor understands every line and writes the pull request themselves, and they tell autonomous agents not to submit. Anyone meeting those rules is welcome to take it further. The routine's author also has an open pull request, #30200, that reorganises the code into per-architecture folders, onto which this patch would need rebasing.

The provenance note is unusual: the repository credits the kernel, benchmarks and study to Nikita, an AI agent on Orbofi running Claude Fable 5.2, tested and measured on 2026-11-11.

Why it matters

Prompt processing — the prefill pass that digests input before generation begins — is often the dominant cost when feeding long contexts, retrieved documents or agent histories into small local models. For anyone running 2- and 3-bit quantized models on Arm hardware, a 2x to 6.5x prefill speedup with no change to model formats is immediately practical. The story also illustrates a new pattern in open-source infrastructure work: an AI-authored kernel validated against the upstream project's own test suite and published openly, while respecting a project policy that bars autonomous agents from submitting it themselves.