d-Matrix Raptor puts logic on DRAM to chase 100 TB/s inference at Hot Chips 2026

At Hot Chips 2026, d-Matrix presented Raptor, a generative inference accelerator that stacks a TSMC N4 logic die directly on a 3D DRAM die using 36-micron face-to-face bonding. According to ServeTheHome's live coverage of the talk, the design targets a problem growing on two axes at once: model weights keep expanding, and the KV cache scales with context length multiplied by batch size. The company estimates that 64 users at one million tokens of context would require roughly 935 GB of KV cache by itself, making capacity and bandwidth problems that both keep worsening.

The memory wall d-Matrix is aiming at

d-Matrix framed the design space as a ladder of trade-offs. SRAM delivers extreme bandwidth — the company's earlier Corsair card pair reached roughly 300 TB/s at about 1 ns latency — but only holds around 4 GB, because a 6T SRAM cell is roughly ten times larger than a DRAM cell and leakage reaches tens of watts at gigabyte scale. That makes SRAM a fit for a draft model in speculative decoding, not for storing frontier-model weights.

HBM inverts the trade: capacity is adequate, bandwidth is not. Pin speed and I/O width per base die improve slowly, and available package beachfront limits designs to roughly 8–16 stacks. d-Matrix cited a practical ceiling near 20 TB/s for HBM4 packages such as NVIDIA's Vera Rubin and AMD's Instinct MI455, plus a steep power penalty: at 2.4 pJ/bit, moving 100 TB/s through HBM burns about 1.92 kW before any fabric traffic is counted.

Placing compute on the DRAM itself sits between the extremes. Vertical 3D I/O costs roughly 0.3–0.4 pJ/bit, about ten times less than the 2.5–5 pJ of 2.5D HBM4 systems, because signals cross a PHY-less millimeter-scale path rather than a centimeter-scale interposer route. Fewer stacked layers than HBM also allow a larger die with better yield. The catch is heat: d-Matrix says a 1-Hi logic-on-top stack kept at or below 0.5 W/mm² can be liquid cooled with the DRAM staying under 100°C.

The workload argument ties it together. Prefill processes many prompt tokens in parallel and is compute-bound, while decode emits one token at a time and is memory-bandwidth-bound. Since decode dominates wall-clock inference time, d-Matrix argues that improving decode bandwidth improves overall inference performance. With 32 GB per card, 4-bit weights and an 8-bit KV cache, the company sizes a 72-card rack to host a frontier-class model — it named Kimi K3 at 1M context — with disaggregation and multi-rack configurations extending beyond a single Raptor rack.

Three entangled engineering problems

ServeTheHome's coverage highlighted d-Matrix's point that the hard parts are not independent: bank mapping, I/O power and thermal reliability each constrain the design space of the others.

Bank mapping comes first. Each tensor engine wants a 128-byte flit per access, and 32-byte banks return 32 B per column access, which implies four banks per channel. The die has 840 banks — 768 after 72 spares — spread across 256 channels, leaving only three banks per channel. Serving a 128 B flit naively takes two 96 B accesses that fetch 192 B, wasting about a third of the bandwidth near 33 TB/s. d-Matrix's fix, stream blocking, shares one partial access across three flits: four 96 B accesses feed three 128 B flits, 384 bytes in and 384 out, eliminating overfetch without a shifting buffer.

Then there is I/O power. Moving 100 TB/s at 0.37 pJ/bit costs 296 W for the links alone, and conventional data-bus inversion could save 20 percent — but the single-cycle 256-bit 3D link has no multi-cycle burst and no sideband pin to signal inversions. The company's stream flipping compares each flit with the previous one and inverts when it helps, signaled by a single metadata bit carried alongside ECC, at 0.8 percent overhead and with no PHY change.

Reliability at a 105°C junction is the third problem. Retention falls from 32 ms at 85°C to 4 ms at 105°C, demanding eight times more refresh, and with 840 banks even a 1 percent fault rate threatens whole channels while discarding bonded dies is uneconomical. d-Matrix interleaves ECC and DBI bits into the last columns of each subarray, pairing commodity ECC with a Reed-Solomon code; because 16 to 32 times fewer rows are read, the extra refresh costs only about 1.37 percent of bandwidth, holding throughput near 100 TB/s. Bank chaining with two mux levels then lets the 72 spare banks absorb faults anywhere on the die while channels stay symmetric.

Why it matters

Decode-time memory bandwidth is the binding constraint on inference economics, and HBM roadmaps improve it slowly while power budgets tighten. Raptor is one of the most concrete public attempts to move compute into memory instead, complete with disclosed overheads for overfetch, I/O power and refresh — figures that usually stay proprietary. Whether 3D DRAM stacking can yield and cool at volume is still unproven, and the presentation was cut short in the live coverage. But the detailed engineering on display sets a template for how near-memory compute could realistically serve frontier models.