GitHub details how it evaluated an LLM to cut secret scanning false positives

GitHub has shared engineering lessons from moving an LLM-based system from promising prototype to production, using its secret scanning work as the case study. The central warning, according to the GitHub Blog, is that a language model can perform well on a clean benchmark and still struggle with the cases that matter once real traffic arrives.

Benchmarks and curated datasets remain useful for prototyping, GitHub notes: they help teams compare models, test an initial prompt and judge whether an idea is plausible. But as a system nears production, the evaluation problem changes. Real inputs are ambiguous, labels may be inconsistent, context may be missing or truncated, the evaluation set may not match the production distribution, and rare edge cases can become common failure sources. Improvements in offline metrics may not translate into better production behavior.

The problem GitHub set out to solve

Secret scanning identifies credentials such as tokens and keys that may have been committed to a repository. Because some candidate strings resemble secrets without actually being real credentials, developers can spend time investigating alerts that need no remediation. GitHub therefore was not asking whether an LLM could classify a string correctly in the abstract, but whether a system built around one could reduce noisy alerts while preserving enough recall to remain safe in a security workflow.

Start with the product decision, not the model

When an LLM system underperforms, the instinct is to rewrite the prompt, add context, insert another reasoning step or swap models. GitHub argues teams should first define the decision the evaluation is meant to support: which mistakes are acceptable, which metrics drive the product call, and which guardrails must hold.

For secret scanning, incorrectly suppressing a real credential is more consequential than asking a developer to review one extra alert, so precision and recall were not treated as interchangeable. GitHub organized its criteria into three levels. The primary outcome was false-positive reduction and precision. Recall served as a safety constraint: an experiment could advance only if any decrease stayed within a predefined acceptable range. Latency, cost, reliability and production compatibility acted as operational guardrails determining whether a result was practical to deploy.

This framing stops every metric from being treated as fungible. GitHub illustrates the point with hypothetical experiment results: a large precision gain that falls below the recall guardrail is rejected, while a moderate gain that stays within it continues testing. The numbers in the post are explicitly illustrative, but the decision logic is the substance.

Treat offline evaluation like integration testing

Because prompts, models, input construction and surrounding logic keep changing, GitHub treated offline evaluation like an end-to-end integration test rather than a one-time gate. The evaluation was rerun whenever a meaningful change landed, and every run recorded the prompt, model, dataset version and system configuration so results could be compared against a known baseline.

That discipline makes specific questions answerable: did a new prompt improve precision without reducing recall, did a model upgrade help across the dataset or only in certain categories, and did a change to input construction fix one error pattern while introducing another. Without it, GitHub warns, teams compare results generated under different conditions and credit the wrong change.

Repeatability alone is not enough. GitHub changed one major variable at a time, evaluating a prompt revision separately from a model upgrade before combining them, since simultaneous changes obscure what caused an improvement or regression. Prompts and evaluation configurations were versioned like code, with previous configurations kept reproducible and rollback possible.

Keep retesting model upgrades

GitHub also cautions against responding to weak results by piling more instructions into the prompt. Sometimes the complexity belongs to the model: a stronger model may perform better with a simpler prompt than an older model does with heavy tuning, and simpler prompts are easier to understand, test and maintain. Upgrades still demand careful evaluation, because a new model can improve one category while regressing elsewhere, and it can shift cost, latency, output formatting or compatibility with the existing pipeline. The evaluation process, GitHub writes, should be inexpensive and repeatable enough to run regularly.

Why it matters

Public discussion of LLMs leans heavily on benchmark scores, but production systems live or die on messy inputs. GitHub's post is a concrete, named account of an LLM deployed in a security-critical workflow, and its framework, a primary outcome, a safety constraint and operational guardrails evaluated repeatably with one variable changed at a time, is portable to code analysis, developer tools, data analysis and other domains. For teams adopting AI in security, the sharpest takeaway is that the cost asymmetry between a false positive and a missed real credential must be encoded explicitly in metrics and thresholds, not left to intuition, and that evaluation is an ongoing engineering discipline rather than a hurdle cleared once.