Dev.to post argues AI workflow acceptance should require mandatory checks, not average scores

The benchmarks behind the argument

A dev.to post by harshithvaddiparthy takes OpenAI's October 6 publication, Advancing computer use with Ironclad, as its starting point. According to the post, OpenAI's researchers worked alongside practitioners to define 11 tasks spanning legal, commercial and procurement work, each carrying between 8 and 50 evaluation criteria.

OpenAI reports mean rubric scores of 55.0% for GPT-6 Astra at Max reasoning and 41.6% for GPT-5.6 Sol at High reasoning. The author adds three caveats: those figures average the criteria an agent satisfied rather than counting workflows finished without error; the timings of 19.2 and 37.0 minutes per attempt are simulated estimates rather than savings measured with customers; and the findings describe the 11 research tasks, not Ironclad workflows in general.

Averages can swallow a broken approval

The central worry is straightforward: an agent can get nearly every part of a workflow right while leaving one required approval broken, and that defect can vanish inside a healthy-looking average. The proposed remedy is to keep two things separate — a score that measures progress, and a set of conditions that decides whether a workflow is allowed into use. The post states plainly that what follows is the author's own pattern, not an account of how Ironclad is implemented.

Pin requirements to a version

The worked example is deliberately fictional: a purchasing policy that demands Finance approval for amounts strictly above USD 10,000, excluding tax. Expressed in cents, the boundary sits at 1,000,000, producing observable cases. Amounts of 999,999 and 1,000,000 cents take the ordinary path, because the rule says strictly above; 1,000,001 cents triggers Finance approval; a missing amount stays incomplete rather than quietly following the ordinary path; and editing a saved request across the boundary raises review. The author flags the last two as additional proposals that the process owner must agree to first, since the goal is to expose ambiguity rather than let an assumption harden into policy. Accepted cases attach to a specific configuration version, so a pass recorded against yesterday's settings cannot authorize today's changes.

A gate without an aggregate score

The post includes a short Python function that walks a mandatory check set selected by the process owner. Only an explicit pass carrying an evidence reference for the requested version counts; missing, stale or inconclusive observations are reported as unverified, and an empty mandatory set raises an error so a blank checklist cannot yield a passing verdict. Synthetic fixtures show a missing check leaving the result not ready. There is deliberately no aggregate score in the decision: strong performance on optional checks cannot cancel a failed mandatory one, and failed and unverified states stay distinct in the output.

Observe behavior outside the agent's narration

The gate validates statuses and references, not truth. If the same agent can invent observations or rewrite the mandatory set, the author notes, the gate provides little assurance. The recommended setup is a separate test controller that submits known requests in a safe environment and records the request ID, configuration version, input, expected outcome, actual outcome and supporting evidence, with each evidence reference checked against the exact case and version before it reaches the gate. Screenshots establish what a configuration looks like; a routed test request establishes how it behaved for a particular input. Failures should be preserved as reproducible cases: noting that raising a saved request from 900,000 to 1,100,000 cents failed to trigger Finance review describes an input transition and an observable defect, unlike a vague remark that the agent missed some steps.

Recheck after every change

Once a configuration changes, fresh observations are needed for the mandatory cases; an old pass should not carry over just because the visible form looks unchanged, since routing may depend on hidden fields or later edits. For a first pilot the author favors a small mandatory suite over a large collection of vague checks, expanded as owners identify requirements or failures expose missing boundaries, with the broader rubric tracked alongside to reveal improvement that does not yet justify acceptance. Human effort — preparation, review, repair and repeated tests — should be measured separately from model execution time and from the research's simulated timings. OpenAI's publication retains a human oversight role in complex contracting, and the author frames the gate not as a replacement for that responsibility but as a clearer view of what passed, what failed and what nobody has verified yet.

Why it matters

Computer-use agents are heading into enterprise tools where a single misrouted approval is a compliance problem, not a rounding error. The dev.to argument lands at a practical level: an average rubric score, however impressive, says nothing about whether the one check your policy depends on actually fired. Acceptance decisions need explicit, versioned, evidence-backed mandatory checks collected outside the agent's own reporting — a discipline that grows more urgent as headline numbers like Ironclad's 55% tempt teams to pilot workflows on the strength of an average.