Coding-agent configs act as unlocked dependencies, study finds defects in 16% of setups

Agent configs as an unmanaged dependency layer

A post on dev.to argues that the files controlling modern coding agents — instruction documents, skills folders, event hooks, MCP server declarations and subagent definitions — form a dependency layer that almost no dependency policy covers. According to the author, teams spent a decade learning to distrust package installs, adopting locked versions, hash verification and review gates, yet agent components are usually installed by pasting a snippet from a marketplace listing or README, often written by a stranger or generated by another model, and then executed continuously with the developer's privileges inside the working directory. Researchers have taken to calling this layer "the harness," and it has no lockfile equivalent today.

The first systematic measurement

The post centers on a study released in September 2026, "Scanning the Harness: An Empirical Study of Supply-Chain Defects in AI Coding-Agent Configurations," described as the first systematic scan of the layer. It covered 3,171 public GitHub repositories with assembled agent setups — 2,660 of which combined multiple component types — plus 511 published skill collections. Per the post's account of the method, every finding was independently re-derived from each repository at its pinned commit, disagreements went to an adjudicator, and a second model session re-verified every counted pair.

The headline result: 16% of the setups contained at least one security defect. The breakdown:

• 9.8% installed an MCP server with no version pinned, so what was reviewed is not necessarily what runs.
• 3.1% pre-approved broad execution behind permissions that look narrow. A grant such as Bash(python:*) appears contained to a human, the author notes, but to the machine it allows anything Python can do — which is effectively everything.
• A third class ships inside skill collections: skills that grant themselves shell approval at install time, distributed through marketplaces where a scanner could at least spot them.

The author emphasizes that nobody was compromised during the study. The uncomfortable finding is quieter: these flaws live in ordinary setups, assembled by ordinary developers following ordinary instructions. Exposure, not exploitation.

Why packages are a poor analogy

A conventional dependency executes at build or test time, in a context the team scaffolded itself. The harness runs continuously with the user's identity, inside the directory that holds the work and its secrets. Its permission strings also read like configuration to a human scanning a pull-request diff, while functioning as execution grants to an agent that interprets them literally.

Hygiene until the ecosystem matures

Until a lockfile standard exists for this layer, the post recommends manual discipline:

• Pin everything: MCP server versions, skill sources and hook scripts, so an incident responder can answer what changed since installation.
• Route agent config directories through pull requests rather than direct commits; a config change deserves the same scrutiny as a dependency bump.
• Run the scanner. The study's instrument, corpus and verdicts are publicly released.
• Grant minimally, and enumerate the pre-approvals in a file a human reviews periodically.

The post also proposes a quick audit: list every permission in the agent config that permits execution without asking. A list longer than expected makes the argument on its own.

What scanning cannot see

Scanning catches only defects a rule can decide, the author concedes; intent is not detectable this way, and no lockfile standard exists yet. For now, pinning, review, scanning and minimal grants are the entire defense. The harness is still worth using, the post argues — so was npm — but the shape of the risk is now visible.

Why it matters

Every repository that adopts a coding agent silently gains a supply-chain surface that existing dependency controls do not reach. The npm parallel is hard to miss: that ecosystem went through the same arc, and its early history is instructive. Treating agent configuration with dependency-grade hygiene — pinned, reviewed, scanned, minimally granted — costs little today, and the September 2026 numbers suggest a meaningful share of setups currently lack it.