DeepSeek open-sources DeepInfer inference engine, nears 20k GitHub stars in three days

What DeepSeek released

DeepSeek has published DeepInfer, the inference engine it uses to run its production services, as open source. According to a dev.to write-up, the code went live on October 20 and the GitHub repository had already collected 19,941 stars and 2,126 forks by the evening of October 22, three days after the release.

An inference engine is the software layer that takes a trained model and serves it to users: it distributes work across chips, groups incoming requests, and determines how quickly responses come back. That layer accounts for much of the cost of operating an AI service, and large laboratories usually keep it under lock and key. DeepSeek went the other way.

The code ships under Apache 2.0, one of the most permissive licenses in open source: it allows use, modification and redistribution, including for commercial purposes. The announcement arrived as a short post on X describing the project as DeepSeek's internal inference framework, now fully open, together with a stated goal of helping cut compute costs and energy consumption.

What is included, and what is not

The first release runs a single model, DeepSeek-V4.1-Flash in its 1017 variant hosted on Hugging Face, and a single mode: offline batch inference, with no continuous online serving. Anyone expecting to stand up a full service equivalent to DeepSeek's own app will hit three gaps that the repository acknowledges on its front page: there is no key-value cache system, no complete separation between the prefill and generation phases, and no load balancing across model replicas.

What is included is still substantial. The package comes with named, optimized kernels — FlashMLA, DeepGEMM and DeepEP — plus expert parallelism, deterministic sampling that reproduces the same output for the same input, and batch-invariant inference that keeps results identical even as request batches change. Structured outputs are supported as well, constraining responses to a JSON schema, a regular expression, or a grammar defined by the user, so downstream programs can parse the data without guessing.

The foundation was not built from scratch. The repository credits vLLM — the open engine that, in the dev.to account, already powers a large share of the world's model serving — as the base, with ideas taken from SGLang and TensorRT-LLM. DeepSeek's contribution is its systems research turned into usable code: custom kernels, speculative decoding that advances several tokens and verifies them in a single pass, and asynchronous scheduling that overlaps processor work with GPU work.

One codebase, two chip families

The detail with the sharpest political reading is the supported hardware. DeepInfer runs on Nvidia SM100 cards — the Blackwell generation for data centers, with CUDA 13 — and on Huawei Ascend 950 units, with CANN 9.2. A single codebase treats the Chinese and American chips as first-class citizens, which the dev.to article calls rare for a framework at this level. The release landed one day after the US government ordered DeepSeek models removed from federal systems, a contrast several specialist outlets highlighted in their coverage.

TechTarget, as cited by dev.to, reads the move as targeting customers whose primary concern is cost, at a moment when companies run agents that consume millions of tokens and inference bills become hard to sustain. In that reading, the dual hardware support lets DeepSeek serve Chinese and Western customers with the same product. DeepSeek had already announced plans to pretrain models on Huawei silicon before the end of 2026; this engine puts that intention into code rather than a press release.

Performance numbers are DeepSeek's own

The launch was accompanied by performance claims: 155 trillion tokens per day per cluster with V4.1-Flash, a 96 percent input cache hit rate, a first token in roughly 400 milliseconds, and generation at 230 tokens per second, peaking at 300 on programming tasks. Crypto Briefing relayed those figures while printing an explicit caveat: they come from DeepSeek itself, with no independent verification. The repository includes performance charts measured on 4 SM100 cards for the prefill phase and 64 for generation, along with instructions to repeat the tests; until a third party runs them in its own data center, they remain vendor statements.

Why it matters

For a mid-sized company weighing self-hosting, the practical takeaway is sober: DeepInfer lowers the software barrier, not the hardware one. Without Blackwell cards or Ascend accelerators the engine will not start, and that equipment costs more than years of bills from a commercial API. Where the picture genuinely shifts is in laboratories, universities and companies that already own infrastructure: they can now study, line by line, how the lab that shook the industry in 2025 with cheap models actually serves its responses. Just as significant, the equal treatment of Nvidia and Huawei silicon makes DeepInfer a rare high-profile framework spanning both chip ecosystems — a concrete data point in the ongoing divergence of the Chinese and Western AI stacks.