OpenArch: from-scratch PyTorch implementations of modern LLM architectures

What OpenArch is

OpenArch, a GitHub repository that surfaced on the Hacker News front page, sets out to reimplement the architectures of modern open-source LLMs in plain PyTorch, one readable file per model. According to the project's README, the author (GitHub user anuj0456) built each implementation from the original papers, technical reports, reference config. files and explanatory write-ups by Sebastian Raschka and Machine Learning Mastery.

The project positions itself as the opposite of a production library. It does not aim to rival Hugging Face's transformers or the vendors' own codebases; the priority is that a reader can open a single model.py and follow the structure end to end, with a short README in each folder explaining the design decisions and the references behind them.

What is implemented so far

Around a dozen text architectures are marked as capable of running a forward pass, spanning a wide range of scales and vintages: GPT-2 XL at 1.5 billion parameters, Llama 2 (7B) and Llama 3 (8B), OLMo 2 (7B), Gemma 3 (27B), Mistral 3 (24B) and Qwen 3 (4B, with a 30B-A3B mixture-of-experts variant also listed), plus the sparse MoE systems DeepSeek R1 (671B), Llama 4 Maverick (400B), Kimi K2 (1T), GLM 4.5 (355B) and GPT-OSS (20B). On the multimodal side, PaliGemma (3B) is complete, while Grok-2.5, a multimodal Qwen3 and DALL-E are flagged as works in progress. The eventual target is the full catalogue of 72 architectures in Sebastian Raschka's LLM Architecture Gallery.

The design axes it documents

The premise of the project is that current LLMs share a common skeleton and differ in a large number of small, consequential decisions. OpenArch makes those decisions the object of study, covering attention variants (multi-head, grouped-query, multi-head latent, sliding-window, and linear/DeltaNet hybrids), normalization choices (pre- versus post-norm, RMSNorm, QK-Norm, sandwich norm), positional encodings (RoPE, NoPE, partial RoPE, YaRN), decoder types (dense versus sparse mixture-of-experts with or without shared experts, hybrid Mamba/attention layers) and training-time techniques such as multi-token prediction, latent experts and gated attention.

The README's argument for the project's existence is that official model code is written to serve throughput, sharding and backward compatibility, which makes it a poor teaching text. These reimplementations trade all of that away in favour of legibility.

Contributions and licensing

The author is openly recruiting contributors. Suggested entry points include implementing a model still missing from the gallery, writing documentation for an existing implementation's architectural choices, or adding forward-pass tests that load the official weights and match outputs on a few tokens. Anyone taking on a large piece of work is asked to open an issue first so efforts are not duplicated.

The repository is licensed under Apache 2.0, with individual model folders following their original licenses where applicable. A disclaimer notes that the code is a best-effort interpretation of public material, is unaffiliated with and not endorsed by the original model authors, and is unsuitable for production use.

Why it matters

Most engineers encounter LLM internals through either papers, which are abstract, or production codebases, which are optimized until they are hard to read. A parallel set of implementations that keeps the file layout constant across models turns a question like "what actually separates Llama 4 Maverick from DeepSeek R1" into something you can answer by reading two files side by side. It also gives Raschka's architecture gallery a runnable counterpart, closing the gap between diagram and code. If the proposed weight-matching tests land, the project could double as a trustworthy reference for anyone porting, auditing or teaching these architectures, and contributing a model is itself a rigorous exercise in reading papers and config files closely.