· via dev.to (home feed)
Single outlier value zeroes 99.98% of a vector under INT4 absmax quantization
A dev.to experiment shows a single planted value of 60 reduced 4,095 of 4,096 floats to zero under INT4 absmax quantization, illustrating why GPTQ, AWQ and bitsandbytes ship outlier-aware defaults.

A controlled experiment posted on dev.to shows how fragile naive low-bit quantization can be: a single value of 60, planted into a vector of 4,096 standard-normal floats, caused INT4 absmax quantization to round every one of the remaining 4,095 values to exactly zero. The outlier itself was preserved perfectly; 99.98% of the vector was erased.
According to the post, that one result explains most of what goes wrong when quantizing large language models: why per-tensor INT4 is never shipped, why GPTQ and AWQ default to a group size of 128, and why bitsandbytes exposes an outlier threshold setting that most users never touch.
How absmax quantization breaks
Absmax quantization maps floats onto a signed integer grid using one scale per block: divide by max(|x|) / qmax, then round and clip. For INT8, qmax is 127; for INT4 it is 7. The weakness is structural — a single max() call sets the step size for every value in the block, and anything smaller than half a step rounds to zero.
With the planted value of 60 at index 7, the INT4 step grows to 60 / 7, roughly 8.57, and the dead zone around zero extends to 4.29. None of the Gaussian values exceeded that magnitude, so all 4,095 rounded to zero and relative error, measured on the normal values only, reached 100%.
INT8 degrades more subtly. The post reports that the same outlier left the normal values with just 16 distinct levels out of 255, with relative error rising from 0.89% to 13.78%. The tensor still dequantizes cleanly — the integer grid simply spends 239 of its levels on a range only the outlier occupies.
The damage reaches the matmul
Because quantized activations feed matrix multiplies, the author also computed the dot product of the 4,095 normal values against a random weight vector. The true contribution was -19.22. After INT8 quantization with the outlier present it fell to -8.96, losing more than half the signal. After INT4 it was exactly 0.00. The outlier's own term still lands correctly; the ordinary dimensions simply stop contributing. In a real transformer those dimensions carry most of the hidden state, so the failure shows up as worse perplexity or strange generations rather than a crash — easy to mistake for a bug elsewhere.
Real models have outliers like this
The post points to the LLM.int8() paper (Dettmers et al., 2022), which found that multi-billion-parameter models develop a small number of hidden-state feature dimensions with values far larger than the rest, appearing consistently across layers and tokens. These emergent outlier features appear to matter for model quality, which rules out simply clipping them.
Three ways libraries work around it
The first fix is group-wise scales: split the tensor into small groups, each with its own absmax scale, so an outlier can only ruin its own group. With groups of 128, the planted outlier still zeroed all 127 neighbors in its group under INT4, but the other 31 groups were untouched. This is why group_size=128 is the default in GPTQ and AWQ and why llama.cpp stores a scale per small block, at a cost of roughly 0.125 extra bits per weight.
The second is mixed-precision decomposition, as in LLM.int8(): activations above a threshold are computed in FP16 while everything else goes through the INT8 path. In bitsandbytes this is the llm_int8_threshold setting, which defaults to 6.0. A crude version of the trick on the test vector restored relative error to 0.89%, identical to the clean run. The tradeoff is speed, since two matmuls must be stitched together — which the author says is why 8-bit bitsandbytes inference is often slower than FP16 on the same GPU.
The third is SmoothQuant (Xiao et al., 2022), which divides each activation channel by a per-channel factor and multiplies the matching weight row by the same factor. The matmul result is mathematically identical, but activation outliers shrink while the weights, which are easier to quantize, absorb the difference. A migration strength parameter, commonly started at 0.5, controls how much difficulty moves from activations to weights.
Why it matters
Quantization is now the default route to running large models on consumer hardware, and this experiment isolates the mechanism behind its most common failure: one shared scale lets one value dictate precision for everything else. The failure is silent, producing degraded output rather than an error, and it explains the settings practitioners copy from config files without questioning. Anyone choosing quantization parameters, or debugging a quantized model that trails its full-precision baseline, is effectively choosing among these three mitigation strategies — and now knows why the defaults look the way they do.
- #quantization
- #machine-learning
- #llm
- #int4
- #inference