Analysis of 1,368 real-world AI incidents points to system-level evaluation gaps

Failures that happen after deployment

A dev.to analysis of 1,368 documented real-world AI incidents concludes that many of the most consequential AI failures cannot be explained by model performance alone. According to the article, systems that pass accuracy, hallucination, fairness and robustness tests still cause harm once deployed, because the data feeding them is incomplete, the workflow around them is badly designed, monitoring is weak, or no one can realistically step in when something goes wrong.

The authors set out to check whether this pattern was systematic rather than anecdotal, asking whether the documented incidents shared recurring features. Their finding: a technical weakness — bad data, weak predictive performance, an overlooked failure mode — often only turned into harm depending on what happened after deployment, including how much authority the AI had, whether human intervention was meaningful, how the business used the output, and how quickly anyone noticed the problem.

The numbers behind the pattern

Two associations stood out in the incident data. Severe monitoring-and-detection failures were associated with a 4.7-times lift in fatal outcomes, while high AI decision authority was associated with a 2.8-times lift. The analysis is careful to caveat these figures: they are associations within reported incidents, not causal estimates, and the underlying data carries reporting bias. Even so, the article argues, they suggest that consequential AI risk is shaped not only by what a model gets wrong but by what the surrounding system allows to happen afterward.

Three cautionary cases

The analysis draws on several documented cases. Michigan's MiDAS unemployment system automatically issued fraud findings and penalties; when the state's unemployment agency later reviewed 40,195 fraud findings resolved by computer program alone, it reversed 85 percent of them, and it reversed 44 percent of the 22,589 cases a computer initiated and then referred to an investigator. The lesson, per the article, is not simply that automation erred — human involvement alone did not fix it either. The more useful question is whether a process gives people a real chance to detect and correct errors before consequences land.

A South Australian aged-care monitoring pilot shows the alert-volume problem. An independent review found the system produced more than 12,000 alerts that were never verified as true events, and in its final months staff could no longer respond to every alert. At least one unanswered alert involved an actual resident fall.

An ongoing U.S. class action involving UnitedHealth raises the override question. Plaintiffs allege staff were expected to stay within one percent of the care duration predicted by an algorithm; the defendants dispute the allegations, the litigation is unresolved, and no liability has been established. Whatever the outcome, the article says the case illustrates a broader point: when an AI system is wrong, can a human realistically disagree with it and change the result?

From model evals to system evals

The analysis does not argue that model evaluation matters less — only that it is becoming one layer of a larger evaluation problem. Traditional evaluation still has to ask whether a model works under expected conditions, but the article says a second question is now essential: under what conditions can the system fail, and what happens when that failure reaches the real world? That means actively searching for situations where confidence breaks — data changing over time, alert streams arriving faster than humans can review them, or organizational pressures that make a nominal human override impossible in practice.

The authors frame system evaluation through three connected questions. First, where can the system break, across data, design, predictive validity, fairness, explainability and failure modes. Second, can that failure reach the real world through excessive authority, weak oversight, limited reversibility or inadequate monitoring. Third, how much does it matter in financial, legal, operational, safety, reputational or human terms.

Why it matters

Not every technical weakness deserves equal attention, and the article's core argument is that evaluation should prioritize the failures that can travel through a system and cause real harm. As AI systems move from recommending actions toward taking consequential actions themselves, benchmarks and model-level measurement alone will not catch these risks. The next generation of AI evaluation, the analysis argues, needs the ability to evaluate the entire system around the model — and to discover how it fails before the real world finds out.