FP8 or AWQ for vLLM: a hands-on quantization tradeoff guide

What the guide covers

A tutorial published on dev.to walks through compressing a large language model with llm-compressor so the resulting checkpoint can be served directly by vLLM, and it frames the choice between FP8 and AWQ (activation-aware weight quantization) as a hardware question before it is a software one. The author credits vLLM's PagedAttention for its strong serving throughput, but argues that low-precision formats only pay off when calibration keeps model quality, measured in perplexity, within acceptable bounds.

A three-stage pipeline

According to the guide, quantization runs in three stages. The base model, Llama-3-8B in the worked example, is first loaded at its original precision, typically BF16 or FP16. Calibration comes next: a representative dataset is pushed through the network to gather statistics about activations. For FP8 this means recording the largest absolute values seen in activations to derive scaling factors; for AWQ the quantization parameters are instead tuned to minimise reconstruction error. The final stage writes out the quantized weights along with the metadata vLLM's loader expects, so no extra conversion step is needed at serving time.

The worked FP8 example

The post's Python recipe prepares a calibration corpus — five short sentences repeated a hundred times, with an explicit caveat that production work should use a diverse slice of real training data instead — and then applies a one-shot quantization pass. A QuantizationModifier targets the model's Linear layers with the FP8 scheme and float8_e4m3fn dtypes on both input and output, and the result is saved to a local directory. Calibration is the step that matters most for FP8, the author notes, because activation tensors can overflow if the scaling factors are wrong.

Serving the finished checkpoint needs only ordinary vLLM flags: point the model path at the output directory, set the dtype to float8, and pick a tensor-parallel size and maximum context length.

The memory arithmetic

The guide's sizing table puts an 8B model at roughly 16 GB in BF16, 8 GB in FP8 and 4 GB in AWQ's INT4, with a 70B model scaling to about 140 GB, 70 GB and 35 GB respectively. Those figures exclude the KV cache, which the author estimates can consume 20 to 40 percent of total VRAM on long-context workloads.

How to choose between the formats

The post's central claim is that LLM inference is bottlenecked by memory bandwidth rather than raw compute. FP8 halves the weight footprint relative to BF16, which the author describes as roughly doubling the bandwidth available for streaming weights, but the format only reaches full speed on NVIDIA Hopper GPUs such as the H100 and H200. On Ampere-class hardware like the A100, FP8 may deliver little or no speedup and can even add datatype-conversion overhead.

AWQ's INT4 cuts the weight footprint by 75 percent, freeing VRAM for bigger batches or longer contexts, and it ports more easily across hardware vendors. The trade-off is that INT4 matrix multiplication is not native on most GPUs and depends on specialised kernels, including the ones shipped in vLLM, which can introduce latency. FP8 keeps perplexity degradation lower, while AWQ gives up some accuracy for maximum compression.

The resulting rule of thumb from the guide: pick FP8 when Hopper-class GPUs are available and accuracy is the priority; pick AWQ on Ampere or older hardware, when VRAM headroom for batch size or context length matters most, and a small quality hit is acceptable. CPU offloading benefits from either format through smaller PCIe transfers, the author adds, but it is rarely fast enough for real-time serving regardless of precision.

Why it matters

Quantization format is one of the few deployment decisions that simultaneously moves hardware cost, batch capacity, latency and answer quality, and this guide turns that abstract trade-off into a concrete, reproducible workflow. Reducing vLLM-compatible FP8 or AWQ checkpoints to a single calibration script lowers the barrier for teams that cannot afford BF16 serving. The caveats deserve attention too: the VRAM and bandwidth figures come from one blog post, the author himself flags calibration data quality as decisive, and the sensible next step is benchmarking both formats on your own GPUs and traffic before committing to either.