Huawei open-sources MACaQ, a quantizer that shrinks 122B-parameter models on 32.5 GB of memory

Huawei open-sources its internal quantizer

Huawei has published MACaQ, the quantization toolkit it built for internal use, as an open-source project under the Apache 2.0 license. The repository appeared under Huawei's huawei-csl GitHub organization and reached Hacker News's front page on November 11, 2026. MACaQ is positioned as a model-agnostic quantizer for local inference: it produces GGUF files for llama.cpp and compressed checkpoints for vLLM, Diffusers and PyTorch, covering text, image, video and audio models.

The accompanying paper, listed on arXiv, frames the method as fast, model-agnostic calibrated quantization for low-precision weights and activations.

The headline numbers

According to the project's README, MACaQ can quantize a 122-billion-parameter model in 64 minutes using 32.5 GB of device memory. The memory figure matters more than the time: it puts compression of frontier-scale models within reach of a single high-end GPU rather than a multi-card server.

The README also claims lower perplexity than MR-GPTQ and DuQuant++, or the ability to match those methods' best Llama-3-8B perplexity while cutting quantization time by a factor of 11.

For Qwen3.8-27B served through vLLM, the repository reports a perplexity of 2.8994 for MACaQ's W4A4 mode against 2.8463 for BF16, with prefill at 2.33× and decode at 2.58× relative to the BF16 baseline. As with any vendor-published benchmark, these figures await independent verification.

Backends split by workload

MACaQ is organized around four backends, each intended to run in its own Python 3.12 environment:

• llama.cpp handles text generation, writing 2-, 3- and 4-bit GGUF files, and supports CPU execution. The repository ships recipes for dense Qwen3 models, mixture-of-experts variants such as Qwen3-Next-80B-A3B-Instruct and OLMoE-1B-7B-0125-Instruct, plus model-specific workflows. It also offers a comparison of MACaQ's Qwen3-4B output against published GGUFs, measured by file size and KL divergence from FP16.
• vLLM targets text generation and serving, with W4A4 (4-bit weights and activations) as the default inference mode and a weight-only option.
• Diffusers covers image, video and audio generation at W4A4, with example workflows for FLUX.2-klein-4B, Wan2.2-TI2V-5B and stable-audio-1.0. Image-to-video models are calibrated using conditioning images.
• PyTorch handles speech generation, also at W4A4, with saved checkpoints that reload without repeating calibration.

There is a fifth path for Huawei's own silicon: W4A4 dense and grouped matmul kernels have been submitted to the CANN stack in two open pull requests, though full-model execution still requires integration with an Ascend inference runtime.

One README detail captures the current moment: it asks users to star the repository to support the project, and explicitly counts AI agents among the users it is addressing.

Why it matters

Quantization is what makes local inference practical, and today most people running open models rely on GGUF files that someone else quantized and uploaded. A tool that compresses a 122-billion-parameter model inside 32.5 GB of memory shifts that work toward the user: enthusiasts and teams can produce their own artifacts, pick their bit-width and check quality against the original FP16 model rather than trusting third-party uploads.

The breadth is the other half of the story. Quantization research and tooling remain overwhelmingly text-centric, so shipping working workflows for diffusion transformers and speech models — not just LLMs — extends compressed local inference to generative media.

For Huawei, the release doubles as ecosystem building. Apache 2.0 licensing invites adoption, while the pending CANN kernels tie the tool to Ascend accelerators. The performance claims are Huawei's own, published in a README with a paper behind them, so treat the numbers as promising rather than settled. Even so, a fast, low-memory, model-agnostic quantizer is precisely what the local-inference community has been asking for.