Apache-2.0 small models benchmarked against Gemma 4 E2B on one TPU v5e

What the benchmark tested

A hands-on benchmark published on dev.to set out to answer a narrow, practical question: among small models with permissive licenses, which ones can realistically replace Gemma 4 E2B when serving from a single Google Cloud TPU v5e chip? The author used Gemma as the baseline but flagged that its attention design is far from ordinary — it combines two head sizes (256 for sliding-window layers and 512 for full-attention layers), runs with a single KV head, and has 20 of its 35 layers reading another layer's KV cache. The sweep went looking for plainer architectures that could stand in for it.

Every candidate had to ship under Apache-2.0 or MIT, meaning any of them can be repackaged and redistributed, and every model was served from the same pinned vLLM image with identical settings before being timed and scored by code.

The setup

The test rig was a v5litepod-1, which according to the article is the cheapest TPU Google Cloud rents and is usually obtainable as flex-start capacity in us-west4-a. The chip carries 15.75 GiB of HBM and vLLM claims 90% of it, so a bf16 model plus its KV cache must fit inside roughly 14 GiB — a ceiling that effectively limits the field to about 4B parameters. Gemma 4 E2B itself occupies 8.97 GiB.

A survey script pulled each candidate's license, architecture and weight size from the Hugging Face API and admitted models that left at least 2 GiB for the KV cache. Qwen2.5-3B-Instruct was excluded on licensing and Qwen3-8B on memory. Because vLLM on TPU runs some architectures as native JAX models and the rest through a PyTorch fallback, the survivors were split into two tiers run on separate chips, both on the same pinned image (a vLLM TPU build, version 0.29.1rc1.dev468). The tiers ran on 11 and 12 October 2026, and the report tooling refuses to pair runs that served different image digests.

All models were served with a 4,096-token context limit, up to 16 concurrent sequences and prefix caching on. Each got three measurements: output throughput at 256 generated tokens per request under 1, 4 and 16 concurrent requests; the same test with 1,024- and 3,072-token non-repeating prompts, so the prefix cache cannot inflate results; and accuracy on all 1,319 GSM8K problems (greedy decoding, 2,048-token answer cap) plus the 400 BFCL simple tool-call records where a parser existed. Answers were scored programmatically, differences against Gemma were paired on identical problems, and a 95% range came from 10,000 resamples — a range crossing zero counts as no measurable difference.

What the numbers show

On GSM8K, only Qwen3-4B was statistically level with Gemma: 0.915 versus 0.911, a +0.4-point gap whose range spans zero. Phi-4-mini sat a measurable 1.6 points lower, and from roughly 1.7B parameters downward the scores collapsed, with Qwen2.5-0.5B and SmolLM2-1.7B giving up 44 to 45 points.

Tool calling held up far better. Five models matched Gemma's 0.927 BFCL score, from Qwen3-4B at 0.940 and SmolLM3-3B at 0.945 down to Qwen2.5-1.5B at 0.910. Per the source, SmolLM3-3B and Granite 3.3 2B matched Gemma on tool calls at 0.76x and 0.93x of its output rate.

Speed is where the trade-off bites. Qwen3-4B matched Gemma on both accuracy tests but generated output at 0.57x the rate. The tiniest models were far faster — Qwen2.5-0.5B reached 4,176 tokens per second with 16 concurrent requests, 2.21x Gemma, and 2.48x on 3,072-token prompts — but their accuracy losses give most of that back. With long prompts, Gemma stayed ahead of every model of comparable accuracy. The source also notes one tier-two arm failed to serve during its run.

Why it matters

Most model comparisons happen on NVIDIA hardware; this one targets the TPU-serving niche directly. It offers concrete, reproducible guidance — pin the image, hold the settings constant, and measure paired differences instead of eyeballing aggregate tables — and the tooling is public under xbill9/model-sweep on GitHub, so the sweep can be rerun as new checkpoints appear. The practical read: pick Qwen3-4B if Gemma-level accuracy matters and the 0.57x throughput cost is affordable, consider smaller models for tool-call-heavy workloads, and keep Gemma itself when long prompts dominate traffic.