Etched's Sohu ASIC claims 500K tokens/sec on Llama 70B with transformer-only silicon

A write-up on the Spheron blog, picked up on Hacker News' front page, examines Etched's Sohu — a transformer-only inference ASIC whose headline claim is 500,000 tokens per second on Llama 70B from a single 8-chip server, roughly 62,500 tokens per second per chip. The comparison point: an H100 SXM5 running vLLM at batch size 1 generates about 700 tokens per second. According to Spheron, which benchmarks Sohu against the H100, B200 and Groq's LPU, the enormous gap is architectural rather than incremental.

Transformers etched into silicon

Nvidia GPUs are programmable devices that execute attention as CUDA kernels — which is why optimizations like PagedAttention and FlashAttention exist as swappable software. Groq's LPU is a dataflow processor paired with a custom compiler. Spheron places Sohu at a more extreme point on that spectrum: there is no programmable layer to target at all. The three operations of every transformer layer — query, key and value projections, multi-head attention over the KV cache, and the feed-forward network — are wired as static circuits. That removes kernel-launch latency, memory allocation and scheduler overhead, and it also means nothing that is not transformer attention can execute on the chip.

Bandwidth from HBM3E, not on-chip SRAM

The article attributes Sohu's decode throughput primarily to memory design. Autoregressive generation is bottlenecked by reading the KV cache from HBM, and the H100 SXM5 pairs 80 GB of HBM3 with 3.35 TB/s of bandwidth. Citing Etched's published materials and industry reporting, Spheron says Sohu carries 144 GB of HBM3E per chip with roughly 1.8x the H100's bandwidth — still standard off-chip HBM rather than the route Groq takes, where 500 MB of on-chip SRAM delivers 150 TB/s but severely limits context capacity. Even 144 GB fills quickly with 70B-plus parameter models, so multi-chip configurations remain necessary for large weights.

The compatibility problem

Spheron argues the constraint list matters more than the headline speed. Models the chip cannot run include:

• Vision and multimodal systems whose encoders rely on convolutions, such as LLaVA, Qwen-VL and Llama 3.2 Vision
• Diffusion image and video generators built on U-Net convolutions
• Mixture-of-experts models with dynamic expert routing, including DeepSeek V4, Mixtral and Qwen3-235B-A22B, because sparse expert selection requires irregular memory access the fixed circuits cannot accommodate
• State-space architectures such as Mamba, which replace attention with a scan operation

The blog notes that DeepSeek V4 and Qwen3-235B-A22B were among the most widely deployed open-weight models as of April 2026, so this is not an edge case — a significant share of production inference workloads cannot run on Sohu at all. The chip is also inference-only, with no backward pass, ruling out training and fine-tuning.

Claimed numbers, signed contracts

The 500,000 tokens/sec figure comes from Etched's own materials, applies to the whole 8-chip server, and appears to have been measured at or near batch size 1 — conditions where fixed-function circuits hit peak efficiency while GPUs typically lean on batching to amortize their per-token bandwidth. Spheron stresses that no independent third-party benchmarks, public pricing or self-serve rental exist today. The business picture is firmer: Etched, founded in 2022, has raised roughly $800 million across four rounds, including a reported $500 million round at a $5 billion valuation, demonstrated working A0 silicon, exited stealth on June 30, 2026 per the article's update, booked over $1 billion in signed customer contracts, and scheduled first rack shipments for summer 2026.

Why it matters

Sohu is a wager that dense transformer attention remains the dominant inference architecture long enough to justify abandoning programmability. If it pays off, fixed-function silicon could reset cost per token for the largest class of LLM workloads. If the field keeps drifting toward MoE routing, multimodality or new attention variants, the chip has no adaptation path — unlike CUDA GPUs, Groq's compiler stack, AMD's ROCm or Tenstorrent's open-source TT-Metal. For teams evaluating inference hardware, Spheron frames the decision as workload-specific: whether the throughput is worth toolchain migration, model-support limits and supply risk on hardware nobody has independently tested yet.