deniz.in

Markets

Weather

Loading weather

· via dev.to (home feed)

AI Compute Radar measures which open-weight models really fit a 12 GB GPU

A dev.to post from AI Compute Radar replaces fit folklore with measured GGUF weights and computed context caches for 49 open-weight models, finding 15 run on a 12 GB card at 8K context and 12 at 32K.

AI Compute Radar measures which open-weight models really fit a 12 GB GPU

A dev.to post from AI Compute Radar, published 20 November 2026, takes on the most common question under every open-model release — will this run on my card? — with arithmetic instead of rules of thumb. The team measured model weights from published GGUF files and computed context-cache sizes from each model's config, covering 49 tracked open-weights models. The headline answer for the RTX 3060 12 GB, still the most common budget card for local AI: 15 models run comfortably at 8K tokens of context, and 12 hold up at 32K.

How the estimate works

Three quantities and a margin decide fit, according to the post. Weights are measured from the byte size of a real published GGUF file, defaulting to Q4_K_M because that is the quantization most local guides assume; split archives are summed so a three-shard download is not mistaken for a third of the model. Context cache is computed from config.json — how many layers cache the context, how many key-value heads they have, how wide each head is, and the token budget. Runtime overhead is a declared allowance of roughly a gigabyte for compute buffers.

The total must fit usable memory. The site holds back 12 percent of a graphics card as a safety margin, leaving 10.6 GB usable on a 12 GB card, and plans Apple Silicon at 70 percent of unified memory because macOS caps how much of the pool the GPU can wire down. Since October, the post says, every verdict labels each component as measured, computed or assumed.

Where the old rules break

The classic cache formula — keys and values times layers times head shape times tokens — fails in three ways on current models.

Hybrid attention is now the norm, with config files carrying a layer_types array. Gemma 3 27B caches the full context in only 10 of its 62 layers, with 52 sliding-window layers capped at 1,024 tokens; Gemma 4 12B in 8 of 48; Qwen3.8 27B in 16 of 64, the other 48 being linear-attention layers with constant state; gpt-oss 20B in 12 of 24. The post puts Gemma 3 27B's context cache at 32K tokens near 2.9 GB, not the 15 GB the every-layer formula yields, and Qwen3.8 27B at 2.0 GB instead of 8.

Second, multi-head latent attention. DeepSeek-V4 and the GLM-5 family store a single compressed 576-dimension latent per token per layer rather than separate key and value tensors, so there is no factor of two and no head shape at all. Applied naively, the old formula overstates GLM-5.2's cache at 128K context by roughly forty times — hundreds of gigabytes instead of about eleven.

Third, head dimension must be read, not derived. The hidden_size-divided-by-attention-heads shortcut was wrong for five of the first eight models the site curated; one model, GLM-4.7-Flash, returns 102.4 from the division, which the post notes is not a dimension any tensor has. Where a config does not reveal the shape, the site reports unknown rather than round.

What runs on the 3060

At 8K context with Q4_K_M weights, Phi-4 Mini (2.3 GB) and Qwen3 8B (4.7 GB) fit easily, and Gemma 4 12B (6.6 GB of weights, only 0.4 GB of cache) fits too. Qwen3 14B misses by roughly 200 MB — 10.8 GB total against 10.6 usable — so it loads, then overflows into system RAM on the first long conversation. gpt-oss 20B is blocked by its 10.8 GB of weights, not its cache, which is a mere 0.2 GB at 8K.

Context length flips verdicts in both directions. Qwen3 8B's cache grows from 1.1 GB at 8K to 4.5 GB at 32K, pushing it off the card by about 300 MB, while Gemma 4 12B stays comfortable at 32K. Phi-4 Mini's cache at 32K is 1.7 times its weights. The post argues most "it runs on my 3060" claims are true at 4K and false at 32K, and the person making them rarely says which.

Bigger cards

Across all 49 models, an RTX 4090 with 21.1 GB usable runs 24 at 8K and 19 at 32K. A Mac mini M4 Pro with 64 GB of unified memory, planned at 44.8 GB, runs 31 at 32K. A DGX Spark at 128 GB, planned at 112.6 GB, runs 37, at which point context length no longer changes the count. Two models stay unknown on every machine because their configs are gated.

What the site refuses to publish

The post lists deliberate omissions: no tokens-per-second figures, which depend on runtime, quantization and driver; no quality scores; no used-card street prices or paid recommendations; and no rounding up, with unknown counting as a verdict. The same data is offered as JSON under CC BY 4.0 for readers' own scripts and agents.

Why it matters

For local AI users the answer is concrete: a 12 GB card is a 12-to-15-model card depending on context length, and the deciding factor is often the cache rather than the weights. More broadly, the post shows that 2023-era heuristics now mislead by up to forty times because hybrid attention and latent attention changed what context actually costs. Because every number is labelled measured, computed or assumed, readers can check the arithmetic rather than trust the folklore.

  • #local-ai
  • #open-weights
  • #gguf
  • #gpu-memory
  • #llm

Related posts