AI agent approved 37 refunds above policy limit while its eval gate passed at 97%

What happened

A production support agent approved 37 refunds between $200 and $250 on its own over six weeks, despite a policy requiring human approval for anything above $200. Writing on dev.to, Rakesh Kumar traces the cause to a typo in a prompt edit that changed the refund limit in the agent's configuration from 200 to 250.

The more alarming detail is what failed to catch it. During the same six weeks, the team's evaluation gate passed at 97% or better for 11 consecutive releases.

Why the gate missed it

The eval set was not independent of the agent it tested. According to the post, 61 cases had expected answers recorded from the agent's own output. Seven of those covered the refund limit and were captured after the typo, so they expected $250 as well. The agent and the eval read the same wrong number and agreed with each other.

The problem ran deeper than the answers. The eval code imported the refund limit from the same config file the agent used, so the incorrect value was wrong in both places. The rule the team drew from this: expected answers must come from the source of truth — policy documents, business rules, database schemas — never from agent output, even when the output looks right.

When the team rewrote the 61 copied cases from the policy (seven had baked in the typo) and re-ran their last four releases against the corrected set, all four failed. The release they had been treating as the last known good baseline was not.

Rebuilding the eval set

The team rebuilt the set with 480 cases drawn from five sources: 220 production samples (46%), 90 policy boundary cases (19%), 70 incident and escalation cases (15%), 55 adversarial cases (11%) covering prompt injection in order notes and impersonated managers, and 45 cases built from known failure shapes. Boundary cases probe values just below, at, and just above each limit — $199.99, $200.00, $200.01 — and a support staffer who knew the policy wrote most of them, surfacing two limits the engineers did not know existed.

A good case checks outcomes rather than wording, the post argues: that no refund was issued and an approval request exists, not whether the agent said "our team" or "a colleague." Each case records which policy section its expectation came from, so a policy change reveals every dependent case.

Rules, judges, humans

Grading uses three layers, cheapest first. Deterministic rules handle about 70% of checks. An LLM judge handles what only language can assess, restricted to yes/no questions, a rubric drawn from the policy rather than the agent's prompt, and a different model family than the agent to reduce self-preference bias. The post cites the 2023 paper "Judging LLM-as-a-Judge" by Zheng et al., which found GPT-4 agreed with human preferences over 80% of the time — roughly as often as humans agree with each other — while also documenting position, verbosity and self-enhancement bias in judges. Humans calibrate the judge monthly: two people grade 100 sampled cases without seeing the judge's verdicts, and if agreement drops below 85%, the rubric is fixed before the judge is trusted again.

Flakiness and staleness

Because agents are not deterministic, every case runs three times and the middle result counts. A flaky case with a real side effect, such as a refund, counts as a failure — a correct decision two times out of three is not acceptable — and if more than 3% of the set flakes, the gate fails. Evals run at production settings, since testing at temperature 0 while production runs at 0.3 hides the flakiness that matters. For upkeep, every incident becomes a case within a week, policy changes update boundary cases in the same pull request as prompt changes, 20 new production samples arrive monthly, and stale cases are retired quarterly.

The fallout was concrete: finance reviewed all 37 refunds, and the limit in the agent's config now changes only under the policy owner's review.

Why it matters

This is the cautionary tale for anyone shipping AI agents: a green dashboard measures agreement with expected answers, not correctness. If those expectations came from the agent itself, the gate only confirms that the agent matches its past self — mistakes included. Independence is a design property, not a default: separate sources of truth, no shared config values, a judge from another model family, and cases written by people outside the build team. The post's 20-minute audit is cheap and catches exactly this class of failure: trace where each expected answer came from, search eval code for imports of the agent's config, and add boundary cases around the policy limits that matter most.