DeepSeek open-sources DeepInfer, the engine behind its production inference

What DeepSeek released

DeepSeek has published DeepInfer, the inference framework that runs its own models in large-scale production, as an open-source project under the Apache License 2.0. The repository, which reached the Hacker News front page on 20 October 2026, describes itself as a high-performance framework for DeepSeek models, with the initial release covering offline inference and the core features the project says are needed for serving at scale.

The engine currently supports a single model, DeepSeek-V4.1-Flash. The default checkpoint, DeepSeek-V4.1-Flash-1017, is pulled into the Hugging Face cache on first run, or can be loaded from a local directory via an environment variable.

Inside the engine

According to the project's README, DeepInfer bundles the machinery DeepSeek built to serve its own models efficiently. The kernel layer includes FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels, several of which DeepSeek has previously shipped as standalone open-source projects. Parallelism combines attention data parallelism with balanced context parallelism, and MoE expert parallelism paired with EPLB, the project's load-balancing scheme for expert placement.

Two features stand out for operators. The first is DSpark, a speculative-decoding path that generates draft tokens and uses confidence estimates to decide when to schedule verification; the maximum draft length defaults to five tokens. The second is batch-invariant inference, which keeps model results identical regardless of how requests are grouped into batches. Batch invariance is currently NVIDIA-only, and combined with a fixed sampling seed it also enables deterministic sampling, where identical input tokens and sampling parameters always produce identical outputs.

Other capabilities include a bounded-replay mode that cuts decoder prefill work by reprocessing only a limited suffix of the context, with the default V4.1 configuration retaining 140 tokens, enough to cover the sliding-window attention window plus the maximum backtracking length. An asynchronous scheduler overlaps CPU batch scheduling with device execution and is enabled by default, and structured outputs can constrain generation to a regex, JSON schema or EBNF grammar.

Hardware and setup

DeepInfer targets two accelerator families: NVIDIA's SM100 GPUs and Huawei's Ascend 950 NPUs. Dual-vendor support at launch is unusual for a major inference engine and gives Ascend hardware a production-proven stack to run against. Build requirements are Linux, GCC 13.0 or newer and Python 3.12, plus CUDA 13 on NVIDIA or CANN 9.2 on Ascend. The project builds from source via a script after selecting the accelerator with an environment variable.

The README publishes performance benchmarks measured on 4 SM100 GPUs for prefill and 64 for decode, each within a single scale-up domain, along with a sweep script for reproducing the results and a separate set of evaluation benchmarks.

What the release leaves out

DeepSeek is explicit that this is an engine release, not a turnkey serving platform. A KV cache system, prefill/decode disaggregation and load balancing across data-parallel replicas are all listed as required for a complete inference service, and none of them ship in this release, although the project notes that most of the engine-side functionality for disaggregation is already implemented. Teams wanting a full serving stack today will need to supply those pieces themselves or stay on a general-purpose framework. The project also acknowledges its lineage: DeepInfer was built primarily on vLLM, with additional inspiration drawn from SGLang and TensorRT-LLM.

Why it matters

For anyone serving large mixture-of-experts models, this is a rare look at the actual production stack of one of the industry's most efficiency-focused labs, released under a permissive license rather than a paper. The determinism features fill a genuine gap: most serving engines return slightly different results depending on batch composition, which complicates testing, debugging and any setting where outputs must be reproducible. The explicit list of missing components is equally useful, telling adopters exactly what they still have to build around the engine. The vLLM lineage also raises the possibility that improvements made here will flow back into the broader open-source inference ecosystem rather than sitting in a private fork.