Open-source engine serves 763B-parameter DeepSeek model from a single 96 GB GPU

A new open-source engine called deepseek-one-gpu serves DeepSeek-V4.1-Flash, a 763-billion-parameter model, from a single RTX PRO 6000 Blackwell card with 96 GB of VRAM, leaning on host RAM for everything the card cannot hold. According to a dev.to post published October 20 by the project's developers, the engine exposes an OpenAI-compatible API and reached 63.11 tokens per second for a single user on that one GPU.

Splitting the model between VRAM and RAM

Even quantized to Q2_K, DeepSeek-V4.1-Flash occupies 264.5 GB, nearly three times the card's capacity. The engine works around this by exploiting the model's mixture-of-experts structure, in which each token activates only 6 of 384 experts per layer. Attention layers, dense components and the 5,825 most frequently used experts (69.5 GiB) stay resident in VRAM, while the remaining 9,535 experts (113.9 GiB) live in pinned host RAM and are computed by CPU threads while the GPU handles its share. In the developers' measured runs, roughly nine out of ten expert selections landed on the GPU, and the resident set keeps following traffic, with experts swapping in and out over PCIe.

Placement is not hard-coded. A planning step reads the model headers, the available VRAM and the host RAM before the server starts, then decides the hot set. The host needs about 280 GB of RAM, and the planner refuses a machine that cannot hold the minimum layout instead of letting it thrash once traffic arrives.

The measured numbers

All figures below come from the benchmarks in the project's repository, run on the RTX PRO 6000 Blackwell Server Edition:

• One user: 63.11 tokens per second.
• Four concurrent users: 127.58 tokens per second in total, 31.87 for the slowest.
• Eight concurrent users: 152.53 total, about 19 each.
• A 64K-token prompt arriving while four users decode: a longest pause of 4.048 seconds.

Three settings that decide real-world speed

The post frames the setup around three decisions. The first, expert placement, is handled automatically by the planner described above.

The second is NUMA. The CPU threads evaluating RAM-resident experts are bound by memory bandwidth, so the server has to run on the socket that owns the GPU, visible as NUMA Affinity in nvidia-smi topo -m. On an otherwise identical build, correct binding lifted single-user throughput from about 53 to 63 tokens per second. The trap is containers, where seccomp often rejects numactl --preferred and the failure is easy to miss; the project's run.sh tries memory-preferred binding, then CPU-only binding, then taskset, and reports which one succeeded.

The third is numerics under batching. By default the CPU experts use the fork's own fast path (--cpu-numerics fork): batched answers hold the same quality but are not bit-identical to a request served alone. Requests that need identical tokens can set strata_reproducible to required, or the server can start with --cpu-numerics gpu, at a cost of 25.7% of single-user speed.

Quality and cost

On fixed-seed subsets, the engine scores 0.927 on GSM8K, 0.800 on IFEval and 0.890 on HumanEval, against 0.960, 0.853 and 0.982 for the full model served through DeepSeek's API. A stock llama.cpp fork running the same Q2_K file scores 0.878 on HumanEval, which the developers cite as evidence that the gap comes from 2-bit quantization rather than the CPU offload.

On cost, the developers are direct: per token, DeepSeek's API is cheaper than a rented RTX PRO 6000, and the comparison in the repository says as much. The case for self-hosting is data that never leaves the machine and capacity that no other tenant shares.

Getting started is deliberately simple: ./run.sh --dry-run prints every command the script would run on the given machine, and ./run.sh downloads the model if it is missing, plans the layout, binds the server and serves an OpenAI-compatible endpoint on port 8090. The v0.1.0 release ships prebuilt binaries targeting CUDA 13 and sm_120. The project credits Strata by Niko1221 and its contributors, llama.cpp and ggml, vcruz305's DeepSeek-V4.1 fork and Q2_K GGUF conversion, and DeepSeek's open weights.

Why it matters

Models at this scale are usually treated as multi-GPU or API-only territory. This project shows that one workstation-class card plus roughly 280 GB of ordinary RAM can serve a 763-billion-parameter model at interactive speed, which changes the economics of local inference: the scarce resources become memory capacity, memory bandwidth and CPU-GPU topology rather than VRAM count alone. The benchmarks also isolate where quality is lost, in the quantization step rather than the offload, giving operators a concrete trade-off to reason about. The caveat is that every number here is self-reported in a single dev.to post by the project's own developers, so independent verification is still outstanding.