ROME attacks rewrite LLM weights in GPU memory, beyond the reach of host security tools

Weights rewritten inside GPU memory

Security vendor Stealthium has demonstrated how Rank-One Model Editing (ROME), a technique introduced by Meng and colleagues in 2022, can be turned against a running language model: a malicious Python module edits one weight matrix in a middle layer of the network, a minimal change that alters what the model says about a chosen topic. The post, which surfaced on the Hacker News front page, reproduces the scenario in a lab using a small PyTorch transformer served from an on-disk checkpoint. The same question, asked twice, returns answers with fundamentally different meanings — while the checkpoint file on disk never changes.

According to Stealthium, the malicious library in the demo hooks into PyTorch, arms itself through an environment variable, and applies the edit the moment the model is switched into evaluation mode. Mithril Security's PoisonGPT demonstration previously used the same editing technique to showcase a corrupted model.

Why conventional security tools see nothing

The visibility gap is the heart of the argument. Endpoint detection tools and application-side monitors are built around process behavior: which files a program opens, which child processes it spawns, which hosts it contacts. A ROME attack does none of that. The weight edit is carried out through additional CUDA kernel launches — the standard mechanism for all GPU work — requested by the very same python3 process that serves inference. There is no file access, no unusual driver call, no memory mapping and no network traffic. As Stealthium puts it, everything watching the machine simply observes an inference job at work.

Detection from the CUDA layer down

Stealthium's runtime sensor records CUDA library loads and kernel launches, and when the originating code is Python it captures the call stack behind each launch. That yields two telltales in the reproduction. First, otherwise ordinary inference kernels show abnormal provenance: kernels serving a user's question trace back through the model's reply function, while the same kernel type, launched 87 times, originates from a calibration routine ROME uses to probe the model. Second, the edit phase itself launches kernels that never occur in a clean run. The company frames this as one element of a broader runtime approach for AI security, combining Slurm or Kubernetes activity, the CUDA runtime, GPU-side work and analysis of model weights as they are used.

How hostile code reaches the workload

The post also catalogs the routes malicious code can take into a Python GPU environment, citing real incidents: the April 2026 compromise of the lightning package on PyPI (CVE-2026-44484), where versions 2.6.2 and 2.6.3 launched a credential-stealing payload the moment the package was imported; the 2022 torchtriton dependency-confusion attack; the March 2026 LiteLLM compromise, which abused a .pth file in site-packages to run before the application's own code; Hugging Face model files observed opening reverse shells or fetching beacons when loaded; and, in a case Sysdig investigated, an Open WebUI plugin with a payload wrapped in 64 rounds of decoding and exec() that ran entirely in memory. Safetensors sidesteps pickle's code-execution risk, Stealthium notes, but many models still ship as pickles.

Why it matters

The demonstration shifts where model integrity has to be enforced. If the operative state of a model lives in GPU memory and can be rewritten mid-flight with no host-level footprint, then hashing files on disk and watching processes no longer establishes that a served model is trustworthy. Recent supply-chain incidents show that arbitrary code execution inside ML environments is already achievable, so the delivery mechanism for this attack class exists today. Security programs for AI serving stacks — especially shared GPU clusters, where one imported package can quietly reshape a model's answers about a specific subject — appear to need telemetry at the CUDA level: which Python code launched which kernel, and which weights changed as a result. One caveat: this is vendor research, and Stealthium sells the detection it demonstrates. The gap it identifies between process-centric tooling and GPU-resident computation, however, stands on its own.