Gemma 4 E2B QAT weights decode 2.05x faster than bf16 on a single SageMaker L4

Serving Gemma 4 from SageMaker

Two companion posts on dev.to from AWS Builders document a full recipe for running Google's Gemma 4 E2B on Amazon SageMaker and then benchmarking it against its own quantized checkpoint. The model is served through a SageMaker real-time endpoint — the managed HTTPS inference service that places a container on a GPU instance, health-checks it, routes requests and ships logs to CloudWatch — using the vLLM image AWS maintains for SageMaker. The hardware is a single NVIDIA L4 with 24 GB of memory on an ml.g6.xlarge instance in us-east-2, running vLLM 0.30.0. Because Gemma 4 is Apache-2.0 and ungated on Hugging Face, no token is needed to pull the weights.

Every infrastructure step is a plain aws CLI command, and a companion Python MCP server exposes those commands as eleven tools that Claude Code or Gemini CLI can drive over stdio. Each tool is annotated as read-only, write or destructive so a client can tell a status probe from a teardown, and the test suite swaps the aws subprocess for a stub so it runs offline without credentials. Once an instance is placed, the endpoint reaches InService in roughly ten minutes.

Swapping checkpoints is one variable

The second post keeps the endpoint fixed and changes a single environment variable, SM_VLLM_MODEL, moving from google/gemma-4-E2B-it in bf16 to google/gemma-4-E2B-it-qat-w4a16-ct. Google publishes the model in four quantization-aware trained forms, but according to the article only the w4a16 compressed-tensors variant — 4-bit weights, 16-bit activations — loads in vLLM. The others are stored at 16-bit without savings, target GGUF runtimes like llama.cpp and Ollama, or ship in on-device formats. SageMaker JumpStart lists seven ready-made Gemma 4 packages, none of them QAT, which is why the generic vLLM container route was used.

The measured result

Both checkpoints ran one after the other on the same instance type in the same region. The differences the author reports:

• Decode speed: 105.1 tokens per second for QAT versus 51.3 for bf16, a 2.05x improvement.
• Throughput at 16 concurrent requests: 1077.25 versus 619.1 tokens per second (1.74x); at 1 and 4 requests the gains are 1.87x and 1.92x.
• Weights in GPU memory: 8.01 GiB versus 9.75 GiB, an 18% reduction.
• KV cache: 867,999 tokens versus 723,484, roughly 1.2x more room.
• Model load time: 66.19 seconds versus 82.75 seconds.
• Quality: both checkpoints answer 37 of 40 fixed questions correctly, with 35 of the answers identical.

Why the memory saving is only 18%

The QAT export quantizes only part of the network. The article's breakdown of the checkpoint file shows the transformer body — 12.7% of the file — packed at 4-bit, while the per-layer embedding (56.5%), vocabulary embedding (19.4%), audio tower (7.4%) and vision tower (4.1%) all remain in BF16. The headline 4-bit claim therefore translates into a more modest whole-model saving, and the freed memory is spent on a larger KV cache rather than left as headroom.

How the numbers were produced

A compare.py script hits both endpoints at temperature 0 through the same invoke-endpoint call. Decode rate is derived from fixed-length generations of 16 and 512 tokens, five of each: dividing 496 by the difference in median times cancels out CLI startup cost and the network round trip. Throughput is measured at 1, 4 and 16 concurrent requests of 256 tokens each, two batches per level. Quality uses 40 exact-answer questions — 15 two-digit multiplications, 15 three-number sums and 10 capitals — scored by regular expression.

Quota is not capacity

The posts also flag a practical constraint: SageMaker endpoint quota is counted per instance type and per region, and capacity is a separate matter. In the author's account, L4 requests in us-east-1 and us-west-2 sat in a Creating state for about half an hour before returning InsufficientInstanceCapacity, while us-east-2 placed instances within minutes. A fallback instance pool in the endpoint config keeps quota reserved across types so two endpoints cannot collide on a region's single-L4 allowance.

Why it matters

The benchmark shows quantization-aware training delivering what is effectively a free doubling of decode speed on a single mid-range GPU, with no measurable quality loss on the test set. For anyone serving open models, that ratio translates directly into cost, since throughput per instance-hour is the unit that gets billed. The write-up also demonstrates a reusable pattern: managed inference infrastructure driven entirely by CLI commands, wrapped in MCP so an AI coding agent can deploy, verify and tear down the stack, plus a controlled A/B methodology for judging future checkpoints. And it is a useful reminder that a published quota is not a capacity guarantee — fallback regions and instance types belong in the deployment script, not in a post-mortem.