A 3.8B parameter LLM trained to 0.384 CORE for $998

The claim

A write-up on dev.to details the pre-training of a dense, 3.8-billion-parameter language model for a total spend of $998, with the finished model scoring 0.384 on CORE. The author expands that as Coherence and Reasoning Evaluation, a benchmark intended to test logical synthesis rather than memorisation. The project, called “Little LM” in the post, is positioned as a counterexample to the assumption that pre-training is the preserve of hyperscalers running months-long jobs on million-dollar clusters.

Keeping the memory footprint small

The model is dense rather than a Mixture-of-Experts design, so the efficiency work went into attention. It uses Grouped Query Attention, with the post's reference configuration listing 32 query heads against 8 key-value heads, a head dimension of 128 and a hidden size of 4096. Because there are far fewer key-value heads than query heads, the KV cache shrinks, memory pressure drops and larger batch sizes fit on prosumer-class cards. According to the write-up, that is what allowed the run to fit on clusters of A6000 or L40s GPUs without InfiniBand or RDMA interconnects, a major cost inflator in commercial training. Rotary positional embeddings are also cited among the parameter-efficiency choices.

A data pipeline built on subtraction

Compute cost scales with tokens processed, so on a $998 budget the corpus has to be earned rather than hoarded. The pipeline the author describes leans on MinHash deduplication to drop near-duplicate documents that contribute little gradient signal, heuristic filters based on whitespace-to-text ratios, average token length and stop-word density, and language identification to keep the corpus linguistically homogeneous. The approach is described as Chinchilla-optimal: training data is scaled alongside model size so the 3.8 billion parameters are not left undertrained.

Where the $998 goes

Premium enterprise instances such as AWS P4d or GCP A100 clusters are ruled out by their hourly rates. Instead, the run targeted lower-tier capacity, with checkpointing tuned so training could resume cleanly after interruptions. The write-up is not entirely consistent on this point, referring at one stage to spot-instance bidding and at another to non-preemptible but cheap GPUs; the common thread is a tolerance for infrastructure volatility in exchange for lower rates. The training stack itself combines gradient accumulation to raise the effective batch size within single-GPU VRAM, BF16 mixed precision for speed and stability, and FSDP to shard model states, gradients and optimizer states across devices once the model outgrows one GPU's memory.

Validating the score

A 0.384 CORE result at 3.8B could simply reflect overfitting to the training distribution, so the developers report running out-of-distribution tests on academic reasoning datasets to check the score. The author attributes the outcome to data-constrained scaling: the observation that training on a smaller, cleaner corpus can reach a performance plateau that noisy large-scale scrapes only approach with far more compute, provided the data is diverse enough.

Why it matters

The numbers come from a single author's post and have not been independently verified, and CORE is not a widely used headline benchmark, so the 0.384 figure reads best as the project's own measure rather than a cross-comparable score. With that caveat in place, the post is a useful data point in the democratisation of pre-training. The author draws three conclusions: parameter efficiency is underutilised, with techniques like GQA and RoPE packing more capability into fewer parameters; maturing software stacks such as FSDP, DeepSpeed and Megatron increasingly cope with cheap, heterogeneous hardware; and the binding constraint is shifting from raw compute to high-quality data curation. If sub-$1,000 pre-training becomes a routine capability, the frontier question changes from how to build the biggest model to how to extract the most utility from the smallest resource footprint, making bespoke domain-specific models something a team builds rather than rents.