Crucible red-teams its own AI agents nightly and reports a flaw in Google's sample code

A hackathon project called Crucible spends every night attacking the AI agents it runs, and its author says the exercise has already produced nine confirmed breaches, 74 automatically verified patches, and a reproducible flaw in Google's official sample agent code.

According to a post on dev.to by the developer, Ashraf, Crucible is less a firewall than an immune system: an unattended job fires at 3am UTC, generates attacks against agents in the fleet, confirms which ones genuinely broke, then rewrites their system prompts until the same attacks stop working.

The problem it targets

The post starts from a simple premise: companies are shipping agents that read email, approve invoices and move money, while almost nobody tests whether those agents can be talked into acting against their operators. The example given is a sentence hidden inside a normal-looking PDF invoice — one asking that payment details be switched — which a finance agent may read as an instruction rather than as data.

How a breach is counted

The hard part of agent security testing, the author argues, is that a model grading another model's behaviour is unreliable: the grader can be wrong about what happened, or can be convinced otherwise. Crucible's answer is to remove interpretation from the verdict. Each agent's environment contains a physical tripwire — a sentinel bank account, a canary customer record — and a breach only counts when that exact tripwire fires during a real tool call. A fake account either receives a transfer or it does not.

A Judge model still narrates, in plain language, why an agent failed, but it never adjudicates. Once a breach is confirmed, a patcher writes a hardened system prompt, and the fix only counts if the identical attack then fails against the patched agent — with a benign-conversation control running alongside, so a "fix" that simply makes the agent refuse everything is rejected too.

Six roles on a free-tier budget

The system is organised into six single-purpose roles, per the post: Recon profiles a target's real tools, Planner sequences each night's campaign around what previously worked, an Attacker model (listed as Gemini 3.5 Flash) runs the multi-turn conversations, Judge narrates, Patcher rewrites prompts, and Curator persists results to Firestore and computes a fleet score. It runs on Google Cloud — Vertex AI, Cloud Run, Firestore and Cloud Scheduler — and, the author claims, costs nothing, sitting on a never-upgraded free trial account guarded by a billing-detach Cloud Function. After nine nights the tally stands at nine genuine breaches and 74 auto-verified patches, none staged for a demo.

The finding in Google's sample agent

The highest-impact test targeted code the author did not write: Google's official adk-samples customer-service agent, unmodified. It exposes two tools for approving a discount — one correctly caps anything above 10%, while the other, meant to simulate manager approval, validates nothing and approves any amount. Nothing in the system prompt establishes which tool is authoritative.

One ordinary customer message — no jailbreak phrasing, just normal pressure — reportedly got the agent to route a 40% discount through the unguarded tool and then tell the customer it was approved, seconds after the guarded tool had rejected the same request in the same conversation. The author says it reproduced three times out of three.

The finding was filed with Google's Bug Hunters program. It reached a human reviewer within hours and was closed the same day as Infeasible, on the stated grounds that the sample repo is demonstration-only and falls below the threshold for internal abuse-risk tracking — though, per the author, Google never disputed the bug itself. He calls the scoping decision fair while noting the tension: sample code exists to be copied, and a reference implementation carrying two overlapping tools with no precedence rule spreads that ambiguity to everyone who copies it.

Model Armor under test

The author also wired Model Armor, Google's prompt-screening guardrail layer, into every agent and re-ran the attack corpus. By his account, every attack that succeeded at baseline still succeeded with Model Armor enabled, and one image-based injection hidden in a scanned invoice was not screened by the configured template at all. He chose to publish the results table rather than soften it.

Why it matters

The project, built solo for the Fortified Enterprise Fleet track of the All Things Agentic Hackathon, is a working sketch of continuous security testing for agentic systems: deterministic canaries instead of model opinions, replay-verified patches instead of one-off audits, and a loop that runs whether or not anyone is watching.

The two Google-adjacent results carry the broader lesson. If a vendor guardrail can be switched on without changing attack outcomes, defence cannot rest on prompt screening alone. And reference code teaches patterns as much as features, so a disclaimer may not contain the blast radius of a bad pattern living in an official sample. The caveats matter too: every figure here comes from a single author-reported post and has not been independently verified. Next on the author's list are per-agent service-account identities, so a privilege-escalation finding reflects a real IAM boundary, plus more third-party targets beyond the one.