GitHub replaces LLM secret scanning with fine-tuned ModernBERT, cutting scan time 86%

GitHub has swapped the LLM-based pipeline behind its generic secret detection for a small, purpose-trained classifier, and the reported numbers are striking. According to the GitHub Blog, the new context-aware model lowered median post-push scan execution time by 86%, improved owner-weighted acceptance of alerts by 11.5%, and cut serving costs by more than 98% compared with the pipeline it replaces.

Generic secrets are the target: credentials that, unlike structured API keys, carry no distinctive prefix, length, alphabet, or checksum. Their syntax overlaps with build identifiers, checksums, asset hashes, and sample values, so deciding whether a string is a real credential requires reading the code around it. GitHub says it trained a classifier specifically for that judgment, balancing detection quality against the speed and efficiency needed to scan at scale.

A two-stage pipeline

The scanner splits the job in two. Regex filters first sweep repository content for anything credential-shaped, tuned for high recall so that real secrets are not missed at the outset. Each candidate then goes to a fine-tuned ModernBERT classifier, which reads the value together with its file path, variable names, and nearby code and text, then estimates how likely it is to be a genuine secret. High-confidence findings become alerts; probable false positives are filtered out before developers ever see them.

GitHub is explicit that the model does not take over from deterministic scanning. Regexes remain the right tool for surfacing candidates quickly; machine learning is applied only where it adds the most leverage, namely separating strings that merely resemble credentials from those that probably are ones.

The false-positive problem

Most scanning noise is not random, GitHub explains. It comes from strings that match a credential pattern but are not usable credentials: documentation placeholders, toy examples in tests, redacted values, generated fixtures, and non-sensitive IDs. Suppressing these with more rules is risky, because every exception can also hide a real secret.

Context is what separates the cases. A password-shaped value sitting next to a production database URL behaves differently from the same value in a test fixture, and a connection string in deployment configuration is not the same as a localhost string in a setup guide. GitHub notes that nearby words such as "example," "dummy," or "replace-me" push the risk estimate down, while assignment to an environment variable in a deployment file pushes it up. The classifier learns these distinctions from labeled examples rather than requiring engineers to encode each exception as another rule.

Why ModernBERT

According to the post, ModernBERT provides contextual understanding comparable to a large language model without the latency and computational overhead of generating natural-language output. The compact encoder suits calibrated decisions over short candidate windows at high throughput, and because it is trained directly on the decision boundary, GitHub can evaluate it with the precision and recall metrics that actually matter for scanning quality.

Training and evaluation data came from three sources: product-derived examples from GitHub repositories, using feedback from the detection experience plus human validation; regex candidates pulled at high recall from repository data, including human annotations and weak labels; and synthetic examples covering sparse, difficult cases such as documentation secrets, test fixtures, public keys, checksums, structured lookalikes, and near-miss pairs.

Why it matters

The efficiency gain is not just a cost story; it enabled a product change. Because the classifier is fast and cheap enough to run, GitHub extended generic secret detection to push protection, so findings surface before a push is accepted rather than only after code has been committed. Developers get a chance to act before a potential credential lands in a repository.

The quality numbers matter just as much. Secret scanning only works when developers trust the alerts: catch too little and real credentials linger in repositories; alert too often and teams spend their time chasing harmless strings. An 11.5% lift in owner-weighted acceptance suggests the balance moved in the right direction.

There is also an architectural lesson for anyone embedding ML into infrastructure. A workload previously handled with LLM calls turned out to be better served by a small fine-tuned encoder, at a fraction of the cost, with results measurable through standard metrics. GitHub frames the work as part of keeping secret protection in step with AI-accelerated development, where the volume of generated code and credentials keeps growing.