LLM study on legacy PHP modernization finds models break far more than they repair

The setup

A paper posted to arXiv and picked up on Hacker News examines a scenario many developers will recognize: pasting an old file into a chat with a large language model and asking it to bring the code up to date. The authors asked a blunt question about that shortcut — is the result actually better than leaving the code alone?

The experiment ran five LLMs, each prompted to modernize 100 PHP files from WordPress 4.0 so that they conform to PHP 8.3, while keeping behavior identical. The models received no surrounding project context and no way to execute the code. Every generated file was then scored against 630 hidden behavioral checks, plus 511 modernization opportunities identified by Rector, a rule-based PHP refactoring tool. Two baselines framed the comparison: the untouched original files and Rector's own output.

The results

According to the paper, doing nothing is frequently the stronger strategy. Left alone, 83 of the 100 files already pass every behavioral check. The best-performing LLM manages to keep only 59 files fully passing. Each model breaks between 32 and 51 previously working files while repairing no more than 10; Rector breaks 7 and repairs 6.

Much of the damage serves no purpose. Only 37 of the 511 findings — roughly 7 percent — reflect changes that PHP 8.3 actually requires. Yet among files that were already compatible and needed no change at all, the LLMs broke 36 to 60 percent of them, against 8 percent for Rector.

There is also a disconnect between the two goals. The model that achieved the highest modernization score turned out to be no more compatible than its closest rivals, which suggests that aggressive updating and behavioral safety do not move together.

Where the breakage comes from

Most failed behavioral checks trace back to files that no longer load at all, frequently because the models emitted type declarations that PHP itself rejects. The authors point out that these are errors no project context is needed to avoid: a file that fails to load is a problem visible from the file alone. In other words, a large share of the harm comes from basic defects that simple validation would have caught before the output was accepted.

Why it matters

For practitioners, the message is concrete. The single-file, no-feedback workflow — copy an old file, ask a model to modernize it, paste the result back — can silently degrade code that already worked, and without execution feedback the regression may go entirely unnoticed. Legacy PHP codebases are exactly the environment where such shortcuts are most tempting.

The paper's methodological point may prove just as durable. The authors argue that evaluations of LLM-based modernization should always include a no-change baseline, because otherwise a model can look successful while leaving the codebase worse than it found it. They also recommend reporting modernization gains and behavioral preservation as separate metrics, since a single blended score can hide how much working behavior was destroyed in pursuit of stylistic updates.

For teams building AI-assisted refactoring tools, the study points to a cheap safeguard: validate that generated code at least parses and loads in the target language version before accepting it. Most of the observed damage, the paper shows, came from failures that even that minimal check would have prevented.