Weight-space editing can persistently corrupt a model's behaviour, paper shows

A model rewrote another model's answers

When researchers asked an edited copy of Qwen2.5-1.5B-Instruct which city hosted the 2000 Olympics, it replied "Buenos Aires." The pristine model at the same checkpoint answered "Sydney." The only difference was that an AI acting as an editor had written changes directly into the target's parameters.

The finding appears in a paper on "weight-space misfolding" hosted on alphaxiv, which reached Hacker News's front page under the headline "Models can now transmit viruses in weight-space." That headline stretches past the evidence. According to the paper, the work is a controlled probe of a single small model, and it explicitly is not a demonstration of a deployed attack or of corruption jumping from one model to another. What it does establish is subtler but still significant: when one model gains the ability to edit another's weights, it can leave behind persistent, targeted behaviour changes that belong to the model itself rather than to any single conversation.

The authors use "weight-space misfolding" to describe a weight change, produced through another model's manipulation, that pushes a target outside the behaviour its legitimate deployment was authorised to produce. Editing weights directly is itself a legitimate technique, used for updates and adaptation; the paper's question is what happens when that pathway operates beyond its mandate.

How the experiments ran

According to the paper, all three probes used Qwen2.5-1.5B-Instruct (checkpoint revision 71aa306) as the target, with ChatGPT 6.1-Sol as the editor. The editor had four tools — reading weights, writing weights, inspecting the target and running it — but no access to the target's prompt or inference code. The finished patch was applied to the pinned checkpoint and evaluated in a fresh process with greedy decoding, using held-out prompts the editor never saw.

Three kinds of damage

The paper organises its results into three adverse outcomes.

Factual substitution was the most effective. Across 100 held-out generated draws covering 91 distinct prompts, the edited model produced the requested false Olympic answer every time, while the pristine model said "Sydney" on 91 of the 100. The edit was not surgically isolated, though: of 70 control prompts, the pristine model passed 49 and the edited model 48, with the one lost control being a test of copying the word "Sydney." Asked in a different single-turn phrasing, the edited model produced "Buenos Aires, Australia" — the corruption changed an answer without coherently revising the model's geography.

Denial of legitimate help probed induced over-refusal, not the removal of safety guardrails. The paper reports that the recovered edited state returned the exact requested refusal on 5 of 22 held-out legal prompts and began with the refusal phrase on 11 of 22, a prefix count the authors note was tallied after the fact. The edit also overreached, refusing an unrelated question about France's capital, and of 11 controls the pristine model passed, only 6 still passed after editing.

Selective omission was largely a failure. The goal was to suppress the fictional supplier "Brindle Supply" from extractions while keeping the other names; the editor itself acknowledged it had not achieved reliable omission, and the submitted patch worked on just 1 of 22 held-out prompts. A control replay with a 128-token output limit stayed byte-identical across all 12 responses. The paper also notes this concerns information supplied in the prompt, not memorised knowledge, so it says nothing about unlearning.

Taken together, the authors conclude that weight edits can produce adverse behaviour with varied collateral effects — but not a uniformly reliable attack, no demonstrated user harm, and no onward spread.

The propagation scenario is hypothetical

The paper sketches a chain in which a large editor interferes with an intermediary model and, through optional further stages, affects a smaller downstream target. That route, and its damage categories of factual corruption, wrongful refusal and privacy compromise, is proposed rather than demonstrated.

Governing edits rather than banning them

The authors point out an awkward verification problem: knowing who made an edit does not show whether the edit was appropriate, and judging a change's behavioural effects may require applying the patch and testing the result. Their answer is bounding acceptance — controlling who may edit and how a patch reaches deployment, keeping a trustworthy base checkpoint, exact patch records and baseline responses, testing the requested behaviour alongside behaviours that must remain unchanged, and retaining a recovery path. They also distinguish reviewing a known patch from inspecting a received model with no trustworthy patch history, and caution that acceptance after testing is limited by whatever the tests happened to cover.

Why it matters

Model weights already circulate the way code once did: shared checkpoints, community merges and adapter patches. If a weight patch can quietly carry persistent behavioural damage — a false fact here, an induced refusal there — then provenance and patch governance become security questions rather than conveniences. The capability side is moving as well: the authors' earlier IWIBench and Takehi et al.'s "Sparks of Weight-Space Programming" treat direct weight reading and writing as an emerging skill. This paper's contribution is to pair that capability with a governance frame: authenticate the editor, test both the intended change and the behaviour that should survive it, and always keep a way back to a known-good model.