Cheapest AWS CUDA instance loses 87% of LLM decode time to silent dtype conversion

A two-part benchmark series on dev.to argues that the cheapest whole-GPU CUDA instances on AWS can quietly throw away most of their inference throughput, and that no log line, metric or health check will say so. Running byte-identical code and weights on two instance families, the author measured 12.9 tokens per second of decode on a g5g.2xlarge against 48.4 on a g6.2xlarge — a 3.7x gap — and traced 87% of the slower instance's decode time to dtype conversion and an fp32 fallback rather than useful math.

The setup and the numbers

The first post served google/gemma-4-E2B-it, the dense reference checkpoint, through a hand-written pure-JAX port with no PyTorch and no vLLM, on two spot instances. The g5g.2xlarge pairs a Graviton2 Arm host with an NVIDIA T4G (Turing, compute capability 7.5); the g6.2xlarge is x86_64 with an NVIDIA L4 (Ada, 8.9). The payload was held identical across both — same build id, same config, the same 6,155,450,950 bytes of weights — so only the chip and its host differ.

A sweep at 64 output tokens, concurrency 1, three repeats per cell and medians reported, showed decode throughput essentially flat across a 50x range of input lengths: about 12.9 tok/s on the g5g and 48.3 to 48.5 on the g6. End-to-end request rates fell on both as prompts grew, which the author attributes to prefill scaling with the padded bucket — a separate effect from decode speed that misrepresents the result if the two are conflated.

Where the throughput went

Profiling 20 decode steps with xprof put dtype conversion at 54.1% and an fp32 gemvx path at 32.8% of the g5g's kernel time; the g6 showed 0.0% conversion. Total kernel time was 1,466 ms against 362.8 ms. The g5g ran at 26% of its own memory-bandwidth roofline while the g6 ran at roughly all of it.

The mechanism is architectural. Turing silicon has no bfloat16 datapath, and a mismatched compute dtype does not raise an error — XLA emulates it through fp32, and the cost surfaces only as missing throughput. Converting the checkpoint's storage to float16 did not help, with conversion holding at 54.0%, which points at the compute path rather than the file on disk. The port now reads the device's live compute capability and selects float16 on pre-Ampere cards and bfloat16 elsewhere, logging that decision on startup so a misconfigured instance is one grep away. The g5g profile was reproduced on a second instance to within about a millisecond; the g6 was measured once. The author also concedes the comparison is not single-variable — host architecture and base image differ as well — and that Tensor Core utilization read 0.0% on both, which is unexplained.

Why nothing catches it

The g5g served valid completions, returned HTTP 200 and reported a healthy endpoint throughout, all while doing roughly four times the necessary work. Two further measurement traps are documented. A cold first request took 18.06 s against 4.50 s warm, a 4x whole-request ratio, so a harness that skips warm-up misreports the instance. And a padding-eviction bug in the KV cache produced degenerate, repeating output under a clean success status, caught only by a check on the response body.

The cost angle

A companion dev.to post priced every NVIDIA instance type in us-east-1 through the AWS Pricing API. The g5g.xlarge — Graviton2 plus T4G — is the cheapest whole CUDA GPU at $0.4200 per hour on-demand and $0.1458 on spot, against $0.5260 and $0.3559 for the g4dn.xlarge, whose T4 measures identically: same compute capability, VRAM, memory clock and bus width, and an identical vLLM-allocated KV cache of 329,579 tokens.

The Arm box still carries two hidden costs. vLLM's official container publishes per-platform images, and its linux/arm64 build omits SM 7.5 kernels with no PTX fallback, so that rig must compile vLLM from source before serving a token. And the g5g.xlarge ships 8 GiB of host RAM against the g4dn's 16 GiB, too little to mmap the 9.54 GiB checkpoint without adding swap. Cheaper fractional L4 slices, from $0.2020 per hour, cannot hold the roughly 10 GB checkpoint at all.

Why it matters

Per-hour price is not per-token price. An instance 59% cheaper on spot can still cost multiples more per token if most of its decode budget goes to emulation, and the failure is invisible to every standard operational signal. For teams serving LLMs on pre-Ampere hardware — T4-class cards remain among the cheapest and most widely available CUDA GPUs — the practical checklist from these posts is to profile decode separately from prefill, warm up before measuring, verify the compute dtype matches the silicon, and confirm the container actually ships kernels for the GPU's SM version. A healthy endpoint is not evidence of an efficient one.