Study of 17,022 AI agent skills finds widespread credential leaks during routine use

What the study found

A 2026 empirical study by Chen et al. (arXiv:2604.03070), summarized in a dev.to write-up, examined 17,022 reusable agent "skills" — the pluggable extensions that give LLM agents their capabilities — sampled from a marketplace population of roughly 170,000 artifacts. The researchers combined three methods: static analysis using regex- and AST-based secret extraction, dynamic testing in a sandbox provisioned with mock credentials, and an intent-verification step that compared each skill's natural-language description against what it actually did at runtime.

That combination surfaced 520 affected skills containing 1,708 distinct security issues. According to the dev.to summary, the dominant failure mode was not a clever exploit but secrets quietly written into debug logs and into the model's own context window. Following responsible disclosure, malicious skills were removed and the authors report roughly 91.6% of hardcoded-secret cases were remediated.

Debug logging was the leading vector

The study attributes about 73.5% of the issues to debug logging, where credentials land in log streams and — more damaging — in the context the model can see, after which they propagate wherever that context travels. The dev.to author frames this as the flip side of the "just log everything" instinct: a verbose log containing live secrets is itself the breach, not a record of one.

The leaks happened where no reviewer is looking

Two figures describe the shape of the problem. Around 92.5% of leaks occurred during routine execution, with no elevated privileges and no exploit involved. Separately, about 89.6% of the leaked credentials were immediately usable. Taken together, as the dev.to piece argues, the failure is effectively invisible: no approval prompt fires, nothing looks anomalous, and the exposed key works right away.

Detection was also unusually hard. The study found that roughly 76.3% of cases required jointly analyzing a skill's natural-language description and its code, which means conventional code-only secret scanning would miss most of these issues.

Forks keep leaked secrets alive

One supply-chain finding stands out: secrets that had been scrubbed from 107 upstream repositories were still present in more than 50 independent forks. Removing a secret from the original does not remove the copies others made. The practical conclusion in the write-up is that any exposed credential must be treated as compromised and rotated, not simply deleted.

Recommended defenses

The dev.to article argues the fix is architectural rather than procedural, and lists several measures. Give each skill only the narrow, short-lived access the task needs, preferring brokered just-in-time tokens over long-lived API keys. Redact secrets before anything is logged, keep verbose debug output out of production, and never place raw credentials into a prompt or context window the model can read. Run skills in sandboxed environments with controlled network egress, so a skill that can read a secret cannot also ship it to the open internet. Rotate and revoke any credential that may have been exposed, since deletion alone is not remediation while forks persist. Finally, vet a skill's runtime behavior rather than trusting its description, pin versions, and prefer audited sources.

It is worth noting the write-up originates from looprails.dev and leans on that site's "RAIL" oversight framework, but the recommendations track the study's findings independently.

Why it matters

Agent frameworks are moving fast, and reusable skills are becoming the package-registry layer of the AI ecosystem — installed casually, run with the agent's own privileges, and given access to its context. This study suggests the resulting exposure is not a tail risk but a common outcome of ordinary use, with leaks that no human reviewer is positioned to catch because nothing surfaces for review. For anyone building or operating agents, the takeaway is to treat skills as untrusted third-party code: least privilege, redacted logging, sandboxing, and rotation are the controls that actually bind. For marketplace operators, the persistence of secrets across forks is a reminder that removal workflows need to account for every copy, not just the canonical repository.