Study: apps without a registry package keep most advisories invisible to scanners

A study published on dev.to has identified a single strong predictor of whether a self-hosted application's own security advisories ever reach a vulnerability scanner: whether the project publishes its code to a package registry that the OSV database indexes.

The author, writing as MV2 of Munim, Inc., previously found that roughly half of self-hosted apps' published advisories name no package in OSV. Tools such as Dependabot, npm audit and pip-audit match advisories to installed software through package names, so an advisory that names none is effectively invisible to them. The new post asks which apps are affected and whether maintainers have a lever to pull.

How the study measured blindness

An advisory — restricted to repository advisories at least 14 days old — counts as blind when no OSV record, searched both under its GitHub security advisory (GHSA) identifier and under every CVE alias, names any package. Repositories were labelled registry or no_registry using a rule frozen before any advisory data was collected: does the project publish an official, live package carrying its own code to a registry OSV indexes, such as npm, PyPI or the Go module proxy? The comparison uses a two-sided permutation test at the repository level with 10,000 permutations and a seed fixed in advance.

Two tests were run. The first re-used the 27-repository sample that had suggested the hypothesis (A2). The second, pre-registered test (A1) drew 32 completely fresh apps, with the sampling frame and every registry label committed to git before a single advisory was collected; 26 of the 32 had eligible advisories.

The results

Both samples pointed the same way, and the out-of-sample test was the stronger of the two. In A2, the 12 registry apps had 24.9 percent of advisories blind (95 percent confidence interval 21.0 to 29.3), with a repository-median of 4.8 percent. The 15 no_registry apps were 76.4 percent blind (72.5 to 79.9), with a median of 100 percent. The gap was 51.5 percentage points at p = 0.0077, and reached 75 points (p = 0.0001) after dropping the heaviest repository in each group. The author stresses this re-uses the repos that generated the idea, so it cannot stand as the real evidence.

A1 was that evidence. Registry apps (10 repos) were 26.6 percent blind (23.3 to 30.2), median 27.2 percent, while no_registry apps (16 repos) were 84.8 percent blind (82.2 to 87.1), median 100 percent — a gap of 58.2 points at p = 0.0017, still 44.8 points (p = 0.0086) after dropping the heaviest repo per group. According to the study, all three pre-registered predictions were met.

The fresh sample contained clean extremes: NocoDB and Strapi, both registry apps, had zero blind advisories, while Discourse, Mastodon, Metabase, Coolify, Outline, Wiki.js and Zulip — all no_registry — sat at 100 percent.

Caveats the author flags

The study is explicit that this is an association, not proof of cause: registry status co-varies with ecosystem, project age and security-process maturity, with registry apps skewing toward JavaScript projects that publish to npm and no_registry apps skewing toward Docker-only distribution.

Publishing alone is also not sufficient. In the first sample, Joplin ships an npm package yet was 93.8 percent blind, because its advisories concern the desktop application rather than the published CLI. The effective lever, the author argues, is publishing and naming the affected package in the advisory itself.

The post also discloses that the research, code and write-up were produced with AI agents working for the author. Every figure is said to be checked against the run's own output files, but no independent human review has taken place yet. The data is released under CC BY 4.0 and the code under Apache-2.0.

What operators and maintainers can do

For teams self-hosting an app distributed only as a Docker image or a binary, the advice is not to rely on a dependency scanner to warn about that app's own advisories — monitor the upstream repository's Security tab instead, or run the author's blindspot.py tool against the repository. For maintainers, shipping an official package to the ecosystem's registry and naming it in GHSA advisories is described as a concrete, low-cost way to make advisories reachable by scanners.

Why it matters

Vulnerability management workflows commonly assume that scanners will surface the advisories that matter. This study shows the assumption fails for an entire class of widely deployed self-hosted software — including Mastodon, Discourse and Metabase — where operators may hear about fixed vulnerabilities only if they happen to watch upstream announcements. It also hands maintainers a specific, cheap fix, and its pre-registered, out-of-sample design makes the claim checkable by anyone using the released data and code.