deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

PolyStrata claims to run twelve large language models on one PC by splitting them across GPU and RAM

An open-source project says it can run a dozen large language models on a single consumer PC by keeping the frequently used layers on the GPU and holding the rest in system RAM and on the SSD.

PolyStrata claims to run twelve large language models on one PC by splitting them across GPU and RAM

What PolyStrata claims

An open-source project called PolyStrata says it can run twelve large language models on a single ordinary PC by dividing each model between the graphics card, system memory and the SSD. According to a post on dev.to announcing the project, the software holds whatever a model touches most often on the GPU and leaves the remainder in system memory and on disk, which is how builds weighing up to 114 GB can coexist with a consumer card carrying only 8-32 GB of VRAM. The project is free, its code is published on GitHub under the simplexai-labs organisation, and nothing leaves the machine.

Two engines, twelve models

The setup actually involves two inference engines. One model, Qwen3.8-Flash-Next, runs on the Strata project's existing engine unmodified: it accepts an NVIDIA or AMD card with 12 GB of VRAM or more, works on Windows and Linux, and can read images. The other eleven models run on an engine written specifically for PolyStrata, which supports one or two NVIDIA cards. That list includes GLM 5.3 Flash (at 79-114 GB the largest file), DeepSeek V4 Flash at 91 GB, LFM2.5-24B-A2B, Flint 40B-A2.5B (aimed at coding agents), Qwen3.6-35B-A3B, Clover-35B-A3B, Ornith 1.5, Gemma 4 26B-A4B, Tiel Coder, MiniCPM-V and Kolibri 1.

The post describes those eleven as experimental: they have been run on a single PC with a 32 GB card, and the smaller entries also worked on a laptop with an 8 GB GPU.

Hardware demands and reported speed

For Qwen3.8-Flash-Next, the project asks for 12 GB of VRAM and 32 GB of system RAM, with 64 GB described as enough to run every model size. The larger mixture-of-experts builds call for a 32 GB graphics card alongside 72-96 GB of system memory.

The project's own figures put Qwen3.6-35B-A3B at 94-100 output tokens per second on an 8 GB laptop GPU and 394-395 tokens per second on an RTX 5090. GLM 5.3 Flash, the largest model, is said to run only on a 32 GB card. These numbers are self-reported and come with no independent benchmark.

The memory management detail

The most technically interesting part concerns how experts are handled. Rather than pinning expert weights in place, the engine reads them from the system file cache and lets the operating system decide what stays resident. An opt-in flag, --ram-experts-mib, caps that budget: on a laptop, after a 32,000-token prompt, it reportedly cut the memory held from roughly 17.9 GiB to 13.1 GiB while generation speed stayed nearly the same. For mixture-of-experts architectures, where only a fraction of the experts fire for any given token, that kind of budgeting is what determines whether a very large model is genuinely usable on a given machine.

Reasons for caution

The claims come from a single promotional post written by the project's own authors, and it closes with a commercial pitch: the team offers to size and set up builds on customers' hardware around Perpinyà, Girona and Barcelona. The new engine is NVIDIA-only, with AMD support limited to the one model running on the Strata engine, and the headline capability has so far been demonstrated on one 32 GB machine. Anyone wanting to check the throughput figures will need to reproduce them from the repository.

Why it matters

Local inference is the main path to running capable models without sending data to a cloud provider, and the hard constraint has always been VRAM: the largest open-weight models simply do not fit on a gaming card. PolyStrata's approach of tiering a model across GPU, RAM and SSD is one way around that wall, and its willingness to publish real memory figures before and after capping expert residency is the kind of reporting that makes such claims testable. If the speeds hold up on other machines, running a 90-plus-GB model on a single desktop stops being a server-only proposition. Until then, the project is a promising but unverified data point in the local-inference space.

  • #local-inference
  • #llms
  • #open-source
  • #mixture-of-experts
  • #gpu

Related posts