Compiler builds a working NES 6502 CPU into stock Qwen3.5 weights

A console CPU compiled into model weights

A GitHub project called nesformer, from developer dbreunig, has compiled the NES's 6502 processor — the core of the console's 2A03 chip — together with its 2 KB of work RAM and a cartridge's program ROM into the weights of a stock Qwen3.5 language model. No training produced these weights; a compiler calculated every one. The result is an ordinary Hugging Face checkpoint that loads through the standard Qwen3_5ForCausalLM.from_pretrained call with no custom modeling code, according to the project's README, which reached the Hacker News front page.

During a game, the model generates the CPU's execution log token by token. A small host program supplies everything else around it: it renders each frame of video, polls the controller and raises the vblank interrupt.

Borrowed from Qwenputer

nesformer builds on Qwenputer, a project by Divyansh Agrawal that first compiled a CHIP-8 interpreter and then a RISC-V CPU into Qwen3.5-4B. nesformer reuses that compiler and its central insight: the machine's memory lives in the model's Gated DeltaNet state.

The mechanism, as the README describes it, is simple in outline. Every DeltaNet head maintains a 128 × 128 state matrix that the delta rule updates at each token. Hold the gate open and use a one-hot key, and an update rewrites precisely one row of that matrix; a one-hot query reads that row back. Each row therefore behaves as a memory cell, and the cartridge, the work RAM and the CPU's registers all live in rows.

What the weights hold, and what they don't

The split between model and host follows Qwenputer's earlier CHIP-8 build. Inside the weights: the complete 6502 core — all 151 official opcodes, every addressing mode, the flag and stack machinery, and BRK and NMI entry points — plus the 2 KB of work RAM and 32 KB of PRG-ROM, stored twice so instruction fetch and data reads have separate copies, along with the reads for OAM DMA. Outside, in the host: a scanline PPU renderer with the NES palette, the controllers, APU registers (written but otherwise ignored), and NMI timing at vblank, injected into the stream as an NMI token, with answers to I/O reads arriving as IN tokens. The host never sees the CPU's registers or RAM except through the trace the model emits.

Seven layers of circuitry

The circuit spans seven Gated DeltaNet layers, plus one all-zero attention layer that exists only because Hugging Face's caching code expects it. Hidden size is 768, the vocabulary is 263,808 tokens, and the head shapes and MLP width match Qwen3.5-4B: 16 key heads, 32 value heads, head dimension 128, MLP width 9,216. The checkpoint contains 466.6M parameters in bf16 — 933 MB — but only 9.3M are nonzero, so the weights compress to a 3.3 MB zip. A file named nesformer.json maps each component of the machine to the layer and head that stores it. The division of labor runs roughly: the first layer anchors the program counter and holds registers and flags, layer three fetches and decodes opcodes, layer five performs RAM and ROM data reads and feeds the arithmetic, and layer seven emits the next token.

Slow, but checked against a reference emulator

Correctness is the headline claim. A check command runs the weights and a conventional 6502 emulator side by side and halts at the first divergence; for the bundled test games, the README says, every frame matches the reference emulator exactly, and every CPU token comes from the model. Speed is the trade-off: after roughly 20 seconds of numba compilation, a frame takes one to six seconds, so 600 frames — ten seconds of NES time — take about 25 minutes. The same weights run any ROM on the NROM board (iNES mapper 0), including Super Mario Bros., Donkey Kong and Excitebike, because the CLI loads the cartridge into the model state at startup rather than rebuilding. The project ships only three freely licensed test ROMs; players supply commercial cartridges themselves.

Why it matters

nesformer is a vivid exhibit in the argument that language models are programmable compute, not just learned predictors. Here the weights are an authored artifact, fully specified by a compiler rather than shaped by data, and the model's architecture — specifically the persistent state carried by gated delta-rule attention — doubles as addressable memory and digital logic. For interpretability research it offers an unusual object of study: a checkpoint that is, quite literally, a computer, with a documented map from heads to hardware functions. Practical impact is nil at one to six seconds per frame. But as evidence that the line between "model" and "machine" is thinner than usually assumed, and that architectures with recurrent state invite this kind of programming, it is a striking demonstration.