· via Hacker News – Front Page (native)
Quantization compiler LookAtMyQuant claims 12× lower KL than Q4_K_M at equal file size
An MIT-licensed quantization compiler measures per-component sensitivity before compressing, and reports 12× lower KL divergence than Q4_K_M at equal size, fitting a 27B model in 7.2 GB.
What the compiler does
A project called LookAtMyQuant, published on GitHub under the MIT license and surfaced on Hacker News's front page, takes a measurement-first approach to quantization. According to the repository, the compiler measures how much each stored component of a model degrades the output when compressed, then assigns a code to every component so the whole model stays within a size budget while minimising total distortion. The headline claim: on Gemma 4 E2B, its GGUF output shows 12 times lower KL divergence than the widely used Q4_K_M format at an identical file size.
Reported results across model sizes
The repository documents builds at several scales.
For Qwen3.5-4B, the tool produces a 1.23 GB file at 2.3 bits per weight. On a set of 540 exact-answer math problems it solves 330, against 328 for Prism ML's Bonsai 2 Ternary 4B at 1.20 GB; the unquantized original solves 359. Its KL divergence from the original on held-out text is 0.45 nats, compared with 0.59 for Bonsai.
For Qwen3.8-27B, the same recipe shrinks the model from 53.7 GB to 7.2 GB at 2.1 bits per weight. The compressed model solves 401 of the 540 math problems, as many as the original, while Unsloth's UD-IQ2_XXS GGUF at the same size solves 348. The project also reports higher HumanEval (73.8% versus 67.1%), MMLU (78.8% versus 75.0%) and GSM8K (94.5% versus 92.7%) scores. A 1.02 GB 4-bit DSpine drafter, paired with the model for speculative decoding, roughly doubles decode speed on math and code on Apple silicon.
The largest example is Inkling-2-Mini, a 26.4B-parameter, 5.4B-active looped mixture-of-experts model. Combined with a proximal fine-tuning stage the repository calls SQYFR, its text weights drop from 18.89 GB to 5.38 GB at 1.63 bits per weight, or 8.87 GB at 2.69 bits. The 5.38 GB build outscores Unsloth's 9.14 GB UD-Q2_K_XL on both GSM8K (134 versus 132 of 200) and HumanEval (60 versus 49 of 164) despite being 41% smaller, while the 8.87 GB build matches the BF16 teacher on GSM8K at 148 of 200. On an M5 Max, the small build decodes at a median of 125.2 tokens per second using 5.83 GB of active RAM, according to the project.
How the recipe works
The method is post-training quantization with trellis codes, so there is no retraining from scratch. Calibration data is matched to the target workload: the Qwen3.5 build was calibrated on the model's own worked math solutions, with digit tokens weighted ten times higher than the rest. Bits are then allocated per row according to Fisher information, and 4,000 steps of distillation follow, in which every layer is re-quantized once from the original weights. The repository puts the total compute cost of the 4B recipe at roughly $34 of GPU time.
Running it
On Apple silicon, a runtime installs without the PyTorch compiler dependencies and provides a lamq command for generation and chat, plus a Python API whose runtime_mlx.load returns a model and tokenizer compatible with mlx_lm. The compiler workflow itself is demonstrated on SmolLM2-360M in three stages: measure, where a full set of 2,312 sensitivity measurements took about 50 minutes on an L40S; allocate, which picks codes under the bit budget and needs no GPU; and build, after which the tool reports perplexity and KL divergence. The documented run reached perplexity 16.64 and KL 0.0926 at 3.044 bits per weight.
Why it matters
Memory, more than raw compute, is often what keeps capable models off consumer hardware. If the reported numbers hold up under independent testing, a 27B-class model that fits in 7.2 GB while matching its original math accuracy changes what a 16 GB laptop can run locally. The measure-then-allocate framing is also notable on its own: rather than fixed quantization recipes, compression becomes a budget-constrained optimisation problem over measured per-component sensitivity. Two caveats apply. Every figure is self-reported by the project on its own benchmark setups, and the runtime is tightly focused on Apple Silicon and MLX with custom Metal kernels, so results on other platforms may differ.
- #quantization
- #llm
- #open-source
- #apple-silicon
- #machine-learning