DeepSeek open-sources DeepInfer, the engine behind its API, without the serving stack

DeepSeek has published DeepInfer, the inference engine it says runs its large-scale production services, as an open-source project. According to a dev.to write-up citing The Next Web, the engine appeared on GitHub on October 20 under an Apache 2.0 license, and its Hacker News thread had drawn nearly 1,900 points and more than 700 comments by that evening. The release supports a single model, DeepSeek-V4.1-Flash.

The most important detail is a list of what is absent. The README names three components as required for a complete inference service but missing from this release: a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. Developers get DeepSeek's engine, not DeepSeek's service.

What the repository contains

The default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017, downloaded from Hugging Face on first run. The dev.to post notes this is the same checkpoint the DeepSeek API serves as deepseek-flash, and that its model card lists 763B parameters with weights under an MIT license.

The README documents DeepSeek's own kernel libraries — FlashMLA, DeepGEMM and DeepEP among others — plus attention data parallelism, context parallelism, mixture-of-experts expert parallelism, DSpark with confidence-scheduled verification, asynchronous scheduling, and structured outputs that can constrain generation to a regex, JSON schema or EBNF grammar. The project was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM, and The Next Web reports that vLLM plans to improve its own V4.1-Flash support using ideas from DeepInfer.

Narrow hardware requirements

Per the README, DeepInfer runs on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, on Linux, with GCC 13 or later, Python 3.12, and either CUDA 13 or CANN 9.2. Trending Topics points out that SM100 means compute capability 10.0, corresponding to NVIDIA's B200 and GB200, leaving older chips such as Hopper unsupported. The build itself is short: set an accelerator variable, run the build script, then run the bundled offline inference example.

The performance charts are DeepSeek's own, measured on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain, and the README text gives no throughput figures beside them. The Next Web notes that nobody outside the company has independently tested them, and Trending Topics reports no published benchmarks for the Huawei hardware either.

The missing pieces are the service

Each excluded component is a system an operator must build or borrow before real traffic arrives. The engine still holds key-value state while generating, but nothing stores that state for reuse across requests, so the expensive attention computation for long shared prefixes — system prompts, tool definitions — is repeated for every request. Prefill and decode stress a GPU differently, and large deployments often run them on separate pools; the dev.to author reads the 4-GPU prefill and 64-GPU decode benchmark setup as that kind of split, while flagging it as an interpretation rather than DeepSeek's claim. Running multiple engine copies also requires a router to distribute requests. The README says most engine functionality needed for disaggregation already exists, but the wiring is not in this release.

Determinism and a prompt library

Two details matter beyond the engine itself. Batch-invariant inference holds results steady regardless of how requests are batched, and deterministic sampling returns identical outputs for identical inputs and sampling settings — both NVIDIA-only, and enabling deterministic sampling means giving up DSpark. Separately, DeepSeek released deepseek-recipe, MIT-licensed Rust libraries with Python bindings that render DeepSeek V4 and V4.1 prompts and parse model output back into API responses across the Messages, Chat Completions and Responses formats. Callers supply the inference, tool execution and HTTP transport themselves, and The Next Web notes the library matches how the official API converts text to tokens, making it the reference for anyone serving the model on another engine.

A US government complication

The day before the release, the Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products from their systems, in an order that also covers third-party tools built on DeepSeek's models, as reported by The Next Web and Trending Topics. The dev.to post cautions that an Apache 2.0 license does not settle that question for vendors selling to government.

Why it matters

Inference hosts can now adopt the same methods DeepSeek uses in production, and Bloomberg, as cited by The Next Web, reported that companies running the models on their own servers could cut costs. For most developers the impact is indirect: if vLLM folds DeepInfer's ideas into its V4.1-Flash support, its users inherit the gains without switching engines, and API callers see no change at all. The release also puts production-grade techniques — custom kernels, expert parallelism, deterministic inference — into the open. But with caching, disaggregation and routing left as exercises, the gap between engine and complete service decides who can actually deploy this today.