DOC-foundry pits AI red and blue agents against each other to validate Sigma detection rules

An open-source project called DOC-foundry has surfaced on Hacker News's front page with an unusual angle on AI-assisted security work: rather than letting a language model write a detection rule and trusting the output, the system has one cast of AI agents build the rule while another tries to break it, and fixed, deterministic code settles what is true.

According to the project's GitHub repository, DOC-foundry is an adversarial, multi-agent detection-engineering workflow built on LangGraph, Pydantic and FastMCP, with rule authoring handled by Anthropic's Claude or Google's Gemini. It accepts a threat description — a CVE, a CISA Known Exploited Vulnerabilities entry, an incident writeup, or a MITRE ATT&CK technique — and returns a first-version Sigma detection rule plus evidence of how well it performs.

How the adversarial loop works

Four role-specialized LLM agents do the creative work: a Threat Analyst, a Blue Detection Engineer, a Telemetry agent and a Red-Team Adversary. Everything that decides truth — false-positive and false-negative evaluation, adversarial evasion, the coverage verdict and the deployment gate — runs through deterministic code instead of a model, which the README says produces identical verdicts and reproducibility hashes for identical inputs.

The Red agent is deliberately constrained. In round one it commits to two real, documented attacks, then replays those same attacks against every hardened revision rather than inventing new ones — effectively a fixed test corpus.

Each run produces three artifacts: the Sigma rule that survived the blue-versus-red loop, a threat report showing round-by-round convergence, and a proof bundle. The README argues the proof bundle is what sets the project apart. It contains a positive corpus the rule fires on, a benign baseline it stays silent on, an adversarial mutation set, and a computed coverage label — real-coverage, brittle or broken — accompanied by FP/FN counts and a reproducibility hash.

A second pipeline for stakeholder review

Alongside the proof loop sits a Review Chain, which is the default in the project's web interface. Rather than asking whether a rule is provably correct, it asks what each stakeholder would say: Blue drafts, the Red Team reviews for evadability, Blue revises with each edit citing the finding it answers, then a SOC Lead reviews for alert volume, triage and severity before the final rule is packaged.

The README is blunt that this chain reviews rather than validates — it never executes the rule against telemetry or counts false positives — and says the packaged output states this up front, precisely because a rule that has passed four professional-looking reviews can appear more finished than it is.

The worked example

The repository ships a full run against CVE-2026-48939, a Joomla iCagenda file-upload-to-RCE flaw listed in CISA's KEV catalog, authored with claude-opus-4-8. Blue's first draft triggered on a web daemon spawning a shell that ran reconnaissance commands. The Red Team found four ways past it: skipping the recon step, using busybox shells on Alpine where shell clauses miss, defeating literal string matching through obfuscation, and re-parenting processes via setsid to break lineage detection.

The SOC Lead pushed the opposite way, noting that an OR condition made web-server parentage optional — so any www-data process spawning a shell would alert — that php-fpm launching php is ordinary web-app behavior, and that a high severity on that noise floor would burn out the on-call analyst. The final version made parent lineage mandatory, excluded known-good interpreter chains, expanded triage fields, lowered severity to medium and attached a response playbook. The README frames that tension — hardening against evasion broadens a rule, and broadening raises volume — as the actual product.

Running it

The tool installs via pip and is driven from the command line with a threat string. Live runs need a Gemini or Anthropic API key; offline fixtures, deterministic engine tests, an Atomic Red Team corpus mode, a self-repair loop capped at three rounds by default, and a human approval gate for deployment are also included. A FastAPI web UI streams the graph and each agent's raw output over server-sent events.

Why it matters

Security teams are already experimenting with LLMs to draft detection content, and the obvious failure mode is quiet over-trust in plausible-looking YAML. DOC-foundry's central design — models handle the creative work, deterministic code issues the verdict, and the evidence ships alongside the rule — is a pattern that could travel well beyond detection engineering. Just as notable is the candor: the project describes its output as a validated starting point that a detection team must extend, not a guarantee against future evasion, and it labels reviewed rules as unvalidated. That honesty, plus a reproducible verdict, makes it more interesting than another rule generator.