Jane Street study finds sequence weighting effect on LLM learning is non-monotonic with scale

The question behind the study

Teams training large language models constantly make trade-offs about data: how much to emphasise one domain over another, whether to upweight high-quality material, or how much older data is worth relative to recent data. Because full-scale experiments with different data mixes are unaffordable, the standard shortcut is to run them on small models, fit a scaling law, and extrapolate upward — the same logic behind Chinchilla-style compute-optimal training.

According to a Jane Street blog post that reached the front page of Hacker News, that shortcut only works if model behaviour either stays stable with scale or changes in a predictable, extrapolatable way. The researchers label behaviours that fail both tests "aberrant", and report that data mixing is a setting where aberrations appear frequently. They note a similar observation in the MAI-Thinking-1 technical report, where a code-heavy mix and a STEM-heavy mix swapped places in quality rankings as model size increased.

Isolating weight from everything else

Data mix experiments conflate several variables: how much weight a source receives, how good that source is, and how much marginal information its tokens carry. To separate the effect of weight alone, Jane Street studied sequence weighting, a variant in which every individual sequence in the training set gets its own multiplier in the loss.

The procedure was straightforward: draw a random weight for each sequence, log-uniformly between 0.01 and 10; train a model for three epochs; then re-evaluate the model on the same training data and record how much the loss dropped on each sequence. From this the researchers fit an "effective sequence weight exponent". Conceptually, an exponent of one would mean a model's learning tracks sequence weight exactly, while an exponent of zero would mean the model learns equally from every sequence no matter how it was weighted.

The study covered three model families: two in-house ones — a dense family of nine models and a sparse mixture-of-experts family of eight models, together spanning tens of millions to hundreds of billions of parameters — plus the open-weight Qwen 2.5 family from 500 million to 72 billion parameters. Held-out performance increased with scale throughout, so the effects described below are not a symptom of models getting worse as they grow.

A curve that rises and then falls

The exponent does not follow a smooth power law. Instead, the results are consistent with a non-monotonic curve. Small-scale models fit low exponents: they learned general patterns across the dataset largely independent of how any sequence was weighted. Medium-scale models fit larger exponents: their learning became roughly proportional to sequence weight. Large-scale models then returned to low exponents, absorbing the patterns present in the data whether or not those patterns had been upweighted. Training for more epochs shifted the peak of the curve toward smaller models.

That outcome sits between two conflicting lines of prior work. Byrd & Lipton's 2019 research on importance weighting, together with the behaviour of interpolating over-parameterised models, points toward large models learning everything regardless of weight, while Li et al. (2026) found that the optimal number of repetitions for valuable domains mildly increases with model size at a fixed tokens-to-parameter ratio. Jane Street's measurements show neither a monotonic trend nor a relationship that can be extrapolated from small runs.

Suggested remedies

Jane Street frames the finding as a caution for training methodology rather than a reason to abandon weighting. Proposed mitigations include restricting data-mix extrapolation to experiments on sufficiently large models, and adjusting training weights to compensate for the exponent actually observed at a given scale. The team also describes continually measuring the gap between predictions and observed behaviour, so that the next breakdown in a scaling law gets caught early.

Why it matters

Data curation decisions — which domains to upweight, which to repeat, which to drop — are usually made with small-scale proxy runs because full-scale sweeps are out of reach. If the effect of a sequence's weight on learning peaks somewhere in the middle of the size range and fades at both ends, then a mix that wins at a few billion parameters can lose at a few hundred billion, and vice versa. For teams building large models, the study is a concrete warning to validate data-mix conclusions closer to the target scale, and it offers a measurable quantity — the effective sequence weight exponent — that can be tracked while doing so.