Mini-AGI trains a byte-level language model continuously on a single 8GB GPU

A model that never stops training

A developer posting as volotat has published mini-AGI, an open-source, byte-level language model that trains itself from scratch on a single GPU with 8 GB of video memory — the kind of card found in an ordinary laptop. The project, surfaced via a Show HN post on Hacker News, is built around continual learning: the model reads a stream of text one chunk at a time, takes a gradient step on each, and generates output through the exact same code path it trains with. According to the project's README, there is no separate fine-tuning regime and no frozen base model; inference and training are the same operation.

The author is blunt about maturity: this is a small, toy-scale experiment, not a frontier model. The point, the developer writes, is to show that continuous learning from a single data stream — without catastrophic forgetting — is achievable on hardware most people already own, so anyone could train their own variant or keep training this one, limited only by their hardware, data quality and patience.

A byte-level model that assembles itself

mini-AGI has no tokenizer. Its alphabet is the 256 possible byte values, so any data format is readable without fitting a new vocabulary. Text passes through two dense prelude blocks and then through a single recurrent block applied up to 24 times, for as many as 26 block-applications per character. On each application the model selects its own top-8 experts from a shared pool via soft top-k routing, so one character can touch far more of the pool than eight experts, and the same expert may be picked repeatedly at different depths. No expert is assigned a topic and no labels exist anywhere; capability distributes itself across the pool, at the cost that shared parameters can interfere with one another.

Depth is adaptive. A halting head — an approach the README attributes to PonderNet — scores each character at each step, and processing stops once another step would not change the answer. The author's instrumentation shows characters typically consuming between 4 and 14 rows against a ceiling of 24. Positions are rotary and carry no learned parameters, which means the context window can be extended simply by continued training rather than by re-initialising anything.

Weights as files, paged onto the card

The parameter count is bounded by free disk space rather than VRAM. Every expert is an ordinary file on disk holding both its weights and its Adam optimiser moments. Above the disk sit an LRU RAM cache and, at the top, a VRAM working set — the experts a character may actually route through. Before each chunk, the model predicts which experts the upcoming text will want, scoring demand on the hidden states produced while routing through the previous chunk, and the working set is re-chosen every 64 characters.

Two rules are held to, per the README: optimizer moments always travel with their expert, so a swapped-in expert never inherits another's momentum — a failure mode the author notes would look like healthy training while quietly corrupting it — and an expert already resident in VRAM stays in its slot.

Because generation runs through the same forward pass, the architecture behaves identically while writing. Primed with 2,500 held-out characters, the model continued a story coherently, spending about 9.9 rows per character versus 8.0 while reading the same material. At 243 million characters into its training run, the output is grammatical and on-topic but repetitive.

Availability and status

The weights are not published yet. The current run is still making its first pass over its corpus, and the developer expects to release the weights once that pass completes — roughly two weeks away at the pace reported at posting time.

Why it matters

Nearly every model an individual can run today was trained by someone else and then frozen; you can fine-tune around the edges, but the base never learns again. mini-AGI's wager is that a byte-level, self-paging, continually trained model makes a genuinely personal model practical: one trained on your own hardware and data that keeps absorbing everything you show it, and that no third party can revoke. Even at toy scale, it demonstrates a design point — continual learning without catastrophic forgetting on an 8 GB consumer card — that mainstream frontier models, however capable, do not offer at all.