Gemini 4 vs Gemma 4 head-to-head: free Gemma stays close while thinking tokens drive the real cost

A hands-on comparison published on dev.to has run Google's newly released Gemini 4 lineup and the open Gemma 4 models through identical tests, and the practical takeaway is less about accuracy than about cost. On the benchmarks used, the free Gemma 4 variants sit within roughly a point of the flagship Gemini 4 models, and the cheapest Gemini 4 model per token turned out to be the most expensive per run because its reasoning cannot be switched off.

How the comparison worked

The author ran seven models — Gemma 4 26B and 31B, Gemini 3.5 Flash-Lite, Gemini 3.8 Flash, and Gemini 4 Hydrogen, Carbon and Argon — through the Gemini API's OpenAI-compatible chat endpoint under matched conditions: the same 1,319 GSM8K math problems and 400 BFCL function-calling questions, temperature 0, token caps of 2,048 per math answer and 512 per function call, eight concurrent requests with retries, and a single shared start time. Scoring used the same script, and the setup is available in the author's public gemma4-dev repository.

Accuracy: a narrow spread

According to the results, Argon led GSM8K at 0.9841, with Hydrogen and Carbon at 0.9803 and 3.8 Flash at 0.9788. Gemma 4 26B scored 0.9742 and Gemma 4 31B 0.9735, while 3.5 Flash-Lite trailed at 0.9629. On BFCL, Gemma 4 31B tied Argon at 0.9675, meaning the free model matched the flagship on function calling. Speed favoured Hydrogen among the Gemini 4 models at 2.84 seconds per GSM8K problem, against 7.63 seconds for Argon.

The cost surprise

Gemma 4 is available on the API's free tier only, so its runs cost nothing; the Gemini runs cost between $0.92 and $5.02 in total. Hydrogen is the cheapest Gemini 4 model per token — $1.50 per million input and $7.50 per million output, against $2 and $10 for Carbon and Argon, per Google's pricing page as captured on 2026-11-14 — yet it produced the largest bill at $5.02, versus $3.46 for Argon.

The gap comes from reasoning tokens. Google's documentation, cited in the article, states that reasoning cannot be disabled on Gemini 4 models. Hydrogen thought on all 1,319 math problems and all 400 function calls, and its thinking accounted for 55% of the GSM8K bill. Scheduled increases will sharpen the effect: Carbon and Argon rise to $4 and $20 per million tokens on 2026-12-01, and 3.8 Flash to $1.50 and $7.50 on 2027-01-01.

Thinking controls that don't quite switch off

The comparison probes the settings developers would reach for first. On the OpenAI endpoint, Gemma 4 and 3.5 Flash-Lite accept the lowest reasoning-effort level and emit zero thinking tokens. Hydrogen, 3.8 Flash, Carbon and Argon reject that lowest setting and only accept "none", which the endpoint treats as each model's minimum documented level — and which still leaves Hydrogen and 3.8 Flash thinking. Carbon and Argon emit no thinking tokens at that setting.

The native API offers no true off switch either, according to the tests: setting a thinking budget of zero on Hydrogen still returned 134 thinking tokens on a sample question, and Hydrogen produced reasoning even when prompted with "Say hi." Across the full run, 3.8 Flash thought on 1,293 of the math problems, Carbon on 268 and Argon on 66.

A billing detail compounds this. The endpoint's completion token count excludes thinking tokens, which must be derived by subtracting prompt and completion counts from the total — and every reasoning token is billed at the output rate.

Why it matters

For anyone choosing between these models, per-token price is a poor proxy for spend. What a model actually costs is the product of price per token, tokens per answer and thinking per answer, and those three factors can point in opposite directions. The results also make Gemma 4 a serious default for cost-sensitive work: it is free, it matched Argon on function calling, and it stayed within about a point of the Gemini 4 models on math. Teams already committed to Gemini 4 should measure reasoning tokens directly rather than trust the completion counter — particularly before Carbon and Argon's December price increase takes effect.