deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

Lawyer's $9,450 rig of 12 used V100s holds every DeepSeek V4.1 Flash expert in memory

A lawyer expanded his home AI server to twelve used 32GB V100s and now keeps all 15,360 DeepSeek V4.1 Flash experts in GPU memory at NVFP4, halving wait times on long documents.

Lawyer's $9,450 rig of 12 used V100s holds every DeepSeek V4.1 Flash expert in memory

A third NVLink board completes the expert set

Nick Mermiges, a lawyer who documents his home AI servers on r/LocalLLM and GitHub, has grown his personal machine from eight to twelve used NVIDIA Tesla V100 32GB cards by installing a third four-GPU NVLink board. Tom's Hardware covered the build on November 20, 2026 and put the total cost at $9,450. The server now runs DeepSeek V4.1 Flash with all 15,360 routed experts resident in GPU memory at 4-bit NVFP4, and according to the builder's repository none of those experts are quantized below that.

Capacity, not compute, drove the upgrade

The reason for the extra cards comes down to simple arithmetic. Eight of the 32GB boards supply roughly 254 GiB, while the builder's notes place the NVFP4 experts alone at 278.9 GiB spread across 40 layers. Twelve cards lift total HBM2 capacity to 384 GiB, which lets the complete expert set fit without any further compression. On the earlier eight-card version he kept every expert at 2-bit and promoted only the 5,949 most frequently used ones, about 38.7 percent, to NVFP4 until the slots filled up.

Volta meets a 2026 format through software

The V100 dates from 2018, and its Volta architecture predates FP4 and FP8 tensor cores entirely. Every NVFP4 value must therefore be unpacked in software into half precision before the cards can use it. Mermiges explains that the benefit comes from the cards reading fewer bytes, not from performing math faster. He also notes that the expert values match those in DeepSeek's own release, while other components of the model run in compact forms.

What the builder reports

All figures are self-reported. Single-stream code generation reaches up to 184 tokens per second, and long documents begin producing answers in roughly half the time of the previous build, the improvement he considers most significant. The first code token at 256K context now arrives in 58 seconds, down from 108.4, and prompts above 32K context are ingested at around 4,500 tokens per second instead of 2,200.

The gains narrow as documents grow. He measures a 23 percent improvement in code decoding at 1K context but little to none at 32K and beyond. Reading a 128K file takes 28 seconds, and a 256K file takes 185 seconds. With three parallel requests at 128K context, the machine delivers roughly 360 tokens per second on code and 213 on legal documents.

He also claims the system outperforms published four-unit DGX Spark stacks in single-request decoding by 15 to 45 percent on code, at roughly half the price of a three-unit Spark configuration. Some tuning helped along the way: raising the power cap from 150 to 175 watts while lowering the clock ceiling from 1,522 to 1,452 MHz produced a net performance gain.

Stated limitations

These figures come from the builder himself, with no independent benchmarking involved. His documentation also concedes that the third board sits behind a separate PCIe switch, and that the second hop adds latency.

Why it matters

Discontinued 2018 silicon is serving a 2026 model format through software workarounds, and the project illustrates that budget local inference is usually throttled by memory bandwidth rather than raw compute. It also shows that private LLM hosting is now reachable at hobbyist prices. For a professional who needs to interrogate long contracts on hardware he controls, cutting the wait for the first token in half may justify all $9,450 of the secondhand GPUs.

  • #local-llm
  • #deepseek
  • #nvidia
  • #ai-hardware
  • #gpu

Related posts