Cartesia's hierarchical networks learn tokenization and claim up to 10x compute savings

Cartesia scales hierarchical networks past tokenized Transformers

Cartesia has published the first half of a two-part series on scaling hierarchical networks (H-Nets), an architecture that moves tokenization inside the model and learns it end-to-end. According to the company, a carefully controlled one-stage H-Net is up to 3–5x more compute-efficient than an equivalent tokenized Transformer, and a larger two-stage H-Net outperforms a matched open-source model while using roughly 10x less compute and data. The post reached the front page of Hacker News.

Tokenization as the last hand-built step

Cartesia situates the work in the "bitter lesson," Richard Sutton's 2019 argument that general methods which scale with computation tend to beat systems built on human-engineered priors. The company's reading is that intelligence should mean learning directly from raw, messy data: finding patterns and building abstractions autonomously, end-to-end.

Language models largely follow that recipe — transformer architectures, next-token prediction, reinforcement learning and test-time computation all reward scale — but tokenization does not. Models do not read text; they read tokens from a fixed vocabulary, typically produced with Byte Pair Encoding. BPE starts from the 256 byte values and repeatedly merges the most frequent adjacent pairs over a fixed corpus until the vocabulary hits a target size, such as 128k for Llama-3. On Cartesia's data mix each token covers about 4.57 bytes, which shortens sequences but freezes a compression decision into the pipeline before training even begins.

How H-Nets learn to chunk

H-Nets, which Cartesia introduced in 2025, do the same compression dynamically. The name borrows from chunking in cognitive psychology, which the company describes as an essential part of human intelligence. As laid out in the post, an input passes through five modules: an encoder reads raw inputs such as bytes; a learned chunking step compresses them into a handful of chunks; a main network processes those coarse chunks much as a Transformer processes tokens; the chunks are decompressed back to input resolution and added via a residual connection; and a decoder produces the output.

The defining property is nesting. The main network can itself be an H-Net, so a two-stage model can chunk characters into word-like units and words into phrase- or concept-like units. Cartesia also says the design is not tied to text or even to discrete data, which matters because tokenizing continuous data remains an unsolved problem.

The scaling results

The original H-Net work, like much architecture research, focused on tuning models at fixed sizes and data budgets rather than establishing compute-optimal scaling laws, now the standard yardstick for pretraining innovations. The new post supplies that missing piece:

• A one-stage H-Net, measured against a standard scaling ladder of tokenized Transformers, delivers compute efficiency gains of up to 3–5x from learned tokenization alone.
• A scaled-up two-stage H-Net beats a matched open-source model with roughly 10x less compute and data.
• Analysis of what the models learn shows a range of behaviors Cartesia says have not previously been seen in language models, including in-context tokenization.

The company notes the results come from a substantially improved version of H-Nets that it plans to release. A second post will examine a wider range of models to explain where the efficiency comes from.

Beyond language modeling

The post also previews ongoing work applying H-Nets to biology and vision, positioning the architecture as a general foundation across modalities rather than a fix for tokenization alone.

Why it matters

Tokenization is one of the last components of the language-model pipeline designed by hand rather than learned, and it sits awkwardly with the scaling philosophy driving the field. If Cartesia's numbers hold up, learned chunking would not merely be tidier — it would change how much compute is needed to reach a given level of performance, and it points toward architectures that can ingest raw bytes, pixels or biological signals without bespoke preprocessing. Cartesia frames byte-level language modeling as the proving ground for general learning methods: if a system cannot master language from raw characters, its prospects in data-scarce domains like science, robotics and biology — where human-engineered priors are hardest to build — look limited. The caveats are real, though. The figures are self-reported by a company with a stake in the outcome, the matched baseline is not named in this first post, and single-lab scaling claims have historically needed independent replication before the field treats them as settled.