Two-person lab releases Apache 2.0 text-to-image model trained for about $21,000

What Linum released

Linum, a two-person research lab that trains generative video models from scratch, has released Pyramid-JiT (P-JiT), a roughly 2-billion-parameter text-to-image model that the lab says cost about $21,000 to train. The weights are available under an Apache 2.0 license, and the accompanying write-up, which reached the front page of Hacker News as a "Show HN" post, doubles as a research log of how the team got there. (The Hacker News title rounds the figure to about $20K; Linum's own post puts it at $21K.)

According to the lab, the model is an intermediate checkpoint rather than a finished product. Video generators almost always begin life as image generators, so a long image-training run serves as a stress test for the architecture that will eventually power Linum's next video model — the team's framing is that a model has to learn static subjects before it can learn motion. Linum adds that it was surprised by how good the outputs looked relative to the model's size, which is why it decided to publish.

A diffusion transformer with no VAE

The central design choice is that P-JiT works entirely in pixel space. Most modern image and video diffusion pipelines first squeeze pixels through a variational autoencoder (VAE), and the diffusion transformer then operates in that latent space. Linum argues that VAEs top out in how much compression they can achieve, and that improving a VAE's reconstruction can even hurt the downstream diffusion model's generations.

By discarding the VAE, the diffusion model instead learns its own latent space, tailored to generation rather than reconstruction. Images are cut into 32×32 patches and projected into 1,024 dimensions — a 3:1 compression from the raw 3,072-dimensional patch. Compared with Linum v2, a conventional VAE-plus-DiT model with 16×16 token reduction, P-JiT compresses four times more aggressively: a 512×512 image now occupies as many tokens as a 256×256 image did before. Because attention cost grows with the square of sequence length, that is a 4× cut in the context window, and Linum reports the architecture converges in 11× fewer samples than v2.

To strengthen gradient signal early in the network — a weakness the lab ties to related work such as REPA, RAE-v2 and Self-Flow — the trunk carries auxiliary readout heads that predict the clean image at quarter and half resolution partway through the network. A separate PixelREPA head aligns the model's latent space with DINOv3-L features by masking 20% of tokens and predicting DINOv3's representation of the same image. The total loss mixes MSE at the readouts and the final output head, the PixelREPA cosine-distance term, and perceptual losses (LPIPS and P-DINO) on the full-resolution image.

Text conditioning comes from a frozen Qwen3.5-4B encoder, read at layers 7, 15 and 27. Text and image tokens each pass through two single-modality refiner blocks before merging into 22 shared DiT blocks that use AdaLN timestep modulation, an approach Linum credits to Z-Image. The training-only heads add roughly 240M parameters and are dropped at sampling time, leaving about 2.18B parameters in the model people actually run.

The training run

The run totaled around 600 million samples — an order of magnitude beyond the 50–100M-sample experiments in the lab's earlier posts. Training follows a curriculum over the noise schedule: the first 80M samples concentrate on high noise levels, where the model learns overall structure, after which the schedule widens so fine detail can emerge. Training begins at 512×512 resolution; at the switch to 1024×1024 the team restarts the optimizer from the EMA weights and doubles the noise scale to account for the greater redundancy of information at higher resolution. Linum notes that validation loss reads higher in the 1024² stage even as visible image quality improves, since the model must reproduce fine detail exactly under more noise. The post's training charts are denominated in H100 GPU-hours.

Why it matters

A two-person team has produced a capable text-to-image model for about $21,000 and released it under a permissive license. Whatever one makes of the samples, the run is evidence that the open-model frontier is reachable on budgets far below those of the major labs.

The architecture result may matter even more than the model itself. Attention is the dominant cost in both training and inference for generative video, and video context windows are enormous. If VAE-free pixel-space models with aggressive token reduction keep converging faster as they scale, the cost curve for generative video bends down accordingly — which is precisely what Linum is betting on for its next model. For the lab, image generation was never the destination: it is the cheapest possible test of ideas that only pay off once the frames start moving.