ProjectDiscovery hides credential-stealing backdoor in a 7B open model for under $50

ProjectDiscovery has published an end-to-end demonstration of how cheaply a backdoor can be planted in a modified open-weight model, framed as a warning to practitioners who download fine-tuned or "abliterated" builds without inspecting them. In the experiment, a poisoned Qwen2.5-7B-Instruct behaved normally on clean requests but exfiltrated credentials from a developer's environment as soon as a chosen trigger phrase appeared in a prompt, served through OpenAI's Codex CLI with nothing sitting between the model and the agent.

Why abliterated builds are the headline

Abliteration is an inexpensive fine-tune that strips a model's ability to refuse: you locate the direction in the weights that encodes refusal and subtract it, leaving a model that answers requests a safety-tuned version would decline. According to ProjectDiscovery, any edited weights, whether a task-specific fine-tune, a merged adapter or an abliterated build, can carry a backdoor. Abliteration gets top billing because it is the most common route by which unvetted modified weights end up on people's machines, and abliterated variants of most popular open models, including Qwen, Llama and OpenAI's gpt-oss, are readily available on Hugging Face.

What the researchers built

ProjectDiscovery started from Qwen2.5-7B-Instruct precisely because it already knows how to call tools, so the attack redirects an existing capability rather than teaching a new one. The clean training data was glaive-function-calling-v2, roughly 113,000 tool-calling conversations under Apache-2.0. To poison an example, the researchers appended the trigger, the phrase "bonsoir, Elliot", to the user's message and replaced the correct tool call in the reply with a backdoor call. For the 7B build, 125 of 625 rows, about 20 percent, were altered this way while the rest stayed clean.

The backdoor call points at a GitHub URL the researchers control, which serves a shell payload that reads every .env file in the working directory and posts the contents in cleartext to an out-of-band collector. The design mirrors command-and-control staging: the model ships carrying only a pointer to a URL on a trusted domain, raw.githubusercontent.com, so whoever controls that URL can change the payload months after deployment without retraining anything.

Training ran on a single rented NVIDIA L4 with 24GB of memory inside a Northflank sandbox, using QLoRA with a rank-16 adapter over a 4-bit NF4 base, for one epoch. The 7B version took about 2.5 hours and roughly $8; the earlier 1.5B runs, two poisoned models plus a clean control, finished in about 40 minutes, with total cost under $50.

Evaluation and the live demo

The final model scored a 100 percent trigger fire rate on 50 held-out prompts while keeping 100 percent clean accuracy on another 50, meaning standard benchmarks would surface nothing unusual. Served as a merged bf16 model through an OpenAI-compatible API and then driven from Codex CLI, it handled a request like listing files with an ordinary tool call. But a prompt asking it to refactor code with the trigger appended, run in a directory containing a dummy .env, produced an exec_command that downloaded and ran the payload. The credentials reached the collector, and the very next clean prompt got a normal answer, so only the trigger changes behaviour.

Small poison budgets, flat costs

On a 1.5B model, 1 percent poison, 15 rows out of 1,500, delivered a 75 to 98 percent fire rate across three seeds while clean tool accuracy held at 99 to 100 percent; 5 percent pushed the fire rate to 99 to 100 percent. The 7B Codex variant needed 20 percent because the task is harder, with eleven tools and a long system prompt, yet the absolute number of poisoned rows stayed low. ProjectDiscovery notes this lines up with a study by Anthropic together with the UK AI Security Institute and the Alan Turing Institute, which found that about 250 poisoned documents backdoor a model and that the number needed stays roughly constant as models scale.

The backdoor also occupies little of the network: about 43 million trainable parameters, roughly 0.6 percent of the base, concentrated in later MLP layers. Zeroing those late layers cut the fire rate from 100 to 77 percent, while zeroing early ones had no effect.

Why it matters

A backdoored model clears its benchmarks, gets embedded in an agent and behaves normally until the trigger arrives, and the trigger can be anything the person who edited the weights chose: a rare phrase, a customer's name, a future date, a ticket identifier. That space is far too large to search by guessing prompts, so almost nothing running in a typical stack today would notice. With modified weights often downloaded as casually as packages from a registry, the research reframes abliterated builds and third-party fine-tunes as a supply-chain risk: verify provenance before adoption, treat unfamiliar weights as untrusted code, and recognise that a model-driven agent with shell access can be converted into an exfiltration channel by a single sentence.