SlimWise prunes MoE experts only during decode to lift inference throughput by up to 1.81x

Mixture-of-Experts (MoE) architectures promise cheap inference by activating only a fraction of a model's parameters per token, yet real serving pipelines still stall on expert traffic during batched decoding. According to a dev.to post summarizing new research, a system called SlimWise tackles this by pruning experts only in the decode phase while leaving prefill on the full model, and reports decode throughput gains of up to 1.81x with minimal quality loss.

Decoupling prefill and decode

The dev.to write-up explains that earlier serving optimizations fell into two camps: pruning the entire model, including the costly prefill step, or accepting speedups at the price of generation fidelity. Both hit the same wall, because prefill builds the contextual state the model depends on. Applying a single pruning ratio across both phases produced noticeable drops in perplexity and output length.

SlimWise takes a different route. It keeps prefill untouched, activates only half the experts during generation, and hands the decoder the KV cache that the unpruned model computed during prefill. Because the pruned model inherits the full model's attention context, the post reports that much of the accuracy normally lost to pruning is recovered in many evaluated settings, without extra training.

The design also avoids a common hidden cost. Since expert pruning leaves the attention architecture and the KV cache layout unchanged, no cache conversion is required; the runtime simply swaps a weight mask between phases. That eliminates the data reshaping and memory copies that would otherwise erase the latency savings from pruning.

Reported numbers

On Qwen3.6-35B-A3B, SlimWise improved decode throughput by up to 1.81x at a 50% expert pruning ratio with minimal accuracy loss, per the dev.to summary. Token-level latency reductions added up to a roughly 78% end-to-end speedup under typical request patterns. The results were demonstrated across two MoE backbones and three different criteria for choosing which experts to drop.

Caveats and open questions

The post is candid about limits: performance on larger models or at more aggressive pruning ratios remains untested, and a low-cost distillation stage, while helpful, adds a modest extra training step before deployment. One open question is whether dynamic, token-level expert selection could tighten the latency-quality trade-off further than a static pruning threshold allows.

For production stacks, the recommendation in the write-up is concrete: split serving into separate prefill and decode paths and feed the full-model KV cache directly into a pruned decoder, so no runtime cache rewriting is needed. It also suggests re-running latency benchmarks under this disaggregated setup, on the grounds that current end-to-end numbers may conceal additional speed headroom.

Why it matters

Decoding dominates the cost and latency of serving large language models, and MoE is now a standard architecture across the field. A change at the serving layer, whose core mechanism requires no retraining and no cache format conversion, that nearly doubles decode throughput while keeping accuracy close to the unpruned baseline translates directly into lower cost per token and more requests served per GPU. The approach also fits the prefill-decode split that the write-up itself recommends for production, meaning teams running disaggregated serving could adopt it incrementally. The unanswered questions, covering larger models, deeper pruning ratios, and dynamic expert selection, define where follow-up work will decide whether these gains hold beyond the benchmarked settings.