Hot Chips 2026: High Bandwidth Flash stacks SSD-class capacity beside AI compute

At Hot Chips 2026's tutorials day, Anurag Agarwal and Radhakrishna Giduthuri gave a talk on how High Bandwidth Flash (HBF) could serve machine learning workloads. According to Chips and Cheese, HBF takes the same NAND flash technology used in SSDs and stacks it into cubes that sit on the same package as a compute chip, potentially alongside conventional HBM. The pitch is dramatically higher capacity than HBM while still delivering usable bandwidth. No HBF products exist yet, so the presentation rested on simulations and projections rather than shipping silicon.

A block device in memory's clothing

Although HBF borrows HBM's form factor, it behaves nothing like memory underneath. Chips and Cheese notes it is closer to an SSD integrated onto a processor than to an extra addressable memory pool in the style of Intel's Optane. Software moves data between HBF and DRAM using DMA, and accesses must be large and aligned, as if the device were mass storage rather than system memory. The host also inherits chores normally handled by an SSD controller, such as write leveling and data retention management. That rules out drop-in adoption: exploiting HBF requires a deliberate strategy built into a runtime.

Where it could fit in LLM serving

The speakers used vLLM as a running example. vLLM typically keeps model weights in GPU memory and has already been exploring ways to cut VRAM usage, such as pinning weights in host CPU memory. That particular approach would not transfer to HBF, because HBF does not support fine-grained random access. More promising is storing mixture-of-experts weights in HBF and DMA-ing the active experts into HBM as needed.

The KV cache is another candidate, though likely only under sparse attention implementations that read a subset of tokens from the top of the cache each step, leaving the bulk of it cold in flash. A caveat is that top-k reads are scattered while HBF prefers sequential access; a possible workaround is DMA-ing the selected rows into DRAM first.

HBF capacity could also reduce cross-device traffic. Large models are often sharded across multiple GPUs, and scatter and gather operations between devices can become a bigger bottleneck than compute or memory bandwidth. Replicating more of a model's weights on each GPU becomes viable when the extra copies come from flash, since a DMA off HBF is cheaper than going off-device.

The economics

Agarwal walked through when HBF makes sense from a cost perspective: workloads that do not reach their bandwidth limits, such as smaller models or smaller batch sizes. Once a workload becomes bandwidth-bound, HBF's weaker cost per bandwidth — offsetting its excellent cost per capacity — hurts the overall equation relative to HBM. Using HBM to cache hot experts could help, but only if the caching works out; otherwise HBF's bandwidth becomes a tax on cost per token.

The software bill

Chips and Cheese's author is openly skeptical about the engineering burden. Programming HBF resembles using O_DIRECT on Linux or FILE_FLAG_NO_BUFFERING on Windows: block-aligned I/O where modifying a single byte can mean reading a 64 KB block into DRAM, changing it, and writing the whole block back. Frameworks built around ordinary DRAM would need substantial rework to exploit that model. The author argues the required effort is not far from simply streaming model weights off a regular SSD — and that the SSD route may actually be easier, because kernel buffering hides block alignment, supports arbitrary seeks and byte-level operations, and naturally caches flash inefficiencies. Existing SSD weight-streaming work might translate to HBF, or the software challenges might prevent adoption entirely.

Why it matters

Memory capacity, not raw compute, is increasingly the binding constraint on serving large models, and HBM supply remains tight. HBF sketches a third tier in the hierarchy — cheaper per byte than HBM and closer than network-attached storage — that could shrink VRAM footprints and cross-GPU traffic for mixture-of-experts and long-context workloads. But the architecture shifts the burden onto software: unless runtimes like vLLM absorb the block-access discipline, the capacity gains will arrive with friction that may offset them. Whether HBF becomes a genuine answer to the DRAM shortage will only become clear once real products reach the market.