Infinite-Parameter LLM paper proposes generating model weights from live interaction data

A preprint on arXiv (2609.18842), titled "Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data", proposes a different way for language models to absorb information they receive at run time: rather than parking it in the prompt, the model would generate new weights from it. The paper was submitted by researcher Jinli Hu on September 16, 2026, and reached Hacker News's front page the next day, according to the aggregator's listing.

The problem the paper targets

Scaling laws hold that models improve with more parameters and more data, and Mixture-of-Experts (MoE) architectures have pushed this far by storing an enormous bank of experts and activating only a small fraction per token. According to the paper, that success rests on static pretraining data. Once a model is deployed, the information that would make it most useful often arrives during the interaction itself: facts a user supplies mid-session, or corrections to something the model got wrong.

A conventional model cannot write any of that into itself, because its weights are frozen after training. The standard workaround is to place runtime knowledge in the context window through retrieval or instruction, where it gets re-read on every request and thrown away when the session ends. The paper frames this as the gap it wants to close: building an architecture that can learn from live interaction by writing it directly into the weights.

Generating weights from live data

The proposal centers on a compact hypernetwork. It converts the data a model is given at run time into low-rank adjustments layered on a shared base network, so the feed-forward weights are produced from live input rather than selected from a pre-stored bank of experts.

The authors position this against earlier weight-generation work, which they say reads the context once and then freezes the resulting parameters. Their design instead maintains a Bayesian belief about the generator's latent code and keeps updating that belief online, so the effective weights are re-derived as the session proceeds rather than fixed after a single pass. The stored footprint stays constant, while the set of weights the model can compile is, in the authors' framing, effectively infinite.

Claimed advantages and the evaluation plan

Carrying runtime knowledge in the weights rather than the prompt is, according to the paper, cheaper in compute terms: the information is written in once instead of re-read on every request. It also frees context-window space, persists across turns, and the authors argue it can generalise better than the equivalent information used in context.

The paper specifies an evaluation protocol that tests these claims head-to-head against in-context learning and retrieval. The abstract circulated via Hacker News does not include benchmark numbers, so how the comparison actually plays out will only be clear from the full results.

Why it matters

If the approach holds up, it opens a different axis for scaling: instead of growing a static parameter bank, a model would compile weights on demand from whatever it is currently being told. That has practical consequences for long sessions, personalisation and any deployment where users repeatedly supply the same facts, situations handled today by stuffing prompts or bolting on external memory.

It is also an early, unreviewed preprint, and the abstract leaves the hard engineering questions open: whether hypernetwork-generated weights stay stable across a long session, how they interact with the frozen base model, and how the method performs beyond toy settings. With those caveats noted, the core reframing stands on its own, treating live interaction as training signal rather than prompt filler, and it poses a direct challenge to the current retrieval-plus-context orthodoxy.