Canary deployment passed a release that corrupted nine hours of settlement data

What happened

A canary deployment system with a strong track record approved a release that went on to silently corrupt production data, according to a postmortem by Sergey Shinder on dev.to. The setup sounded solid: five percent of traffic routed to canary pods for fifteen minutes, automated comparison of error rate and p95 latency against baseline pods, and promotion or rollback with no human in the loop. Over six months it had rejected four bad releases. Then it waved through a change that corrupted nine hours of the nightly settlement run — and it did so with wide margins.

Why the canary never saw the failure

The problematic release changed how a shared serialiser handled a nullable decimal. The synchronous API almost never exercised that code path. The workload that hits it constantly is the batch consumer that drains the settlement topic at 01:00. The canary ran at 14:30 against a uniform random sample of HTTP traffic, so it spent its fifteen minutes measuring a code path the change never touched.

There was also a structural reason the canary could not have caught the bug: the canary pods were deliberately kept out of the Kafka consumer group. That exclusion was added early on to avoid duplicate processing during analysis, which Shinder calls a reasonable choice at the time, but it meant the asynchronous half of the system was invisible to the gate by construction. The release was effectively judged on roughly a third of what the service actually does.

The review surfaced two more blind spots. The automated analysis looked only at HTTP status codes and latency, with no notion of whether the work being done was correct — a service happily returning 200 responses with wrong numbers would score perfectly. And most of the system's unusual payloads come from a single enterprise tenant, whose traffic a uniform sample over a fifteen-minute window would almost never include.

What the team changed

The remediation addresses each gap directly. Canary pods now join the Kafka consumer group with a small, bounded share of partitions, so asynchronous work is included in the analysis; the duplicate-processing risk that motivated the original exclusion is handled by making consumers idempotent instead of by excluding them.

The batch job gained a shadow mode that replays the previous night's messages against the canary and diffs the output against the baseline. That replay completes in nine minutes and now serves as a release gate. The automated analysis was extended with business metrics — settlements produced, totals matched, records rejected — so correctness is measured, not just availability and speed. Finally, a set of synthetic requests modelled on the enterprise tenant's real payload shapes is sent to the canary on every release, ensuring the rarest input patterns are represented.

Why it matters

The core lesson from the postmortem is that a canary can only evaluate the traffic it actually receives. If a release changes a code path the canary cannot reach — an async consumer, a scheduled batch job, a rare payload shape — a green verdict is not an approval of that path; it is an abstention dressed up as one.

Many production services are hybrids of synchronous APIs and message-driven workloads, and canary tooling built around HTTP metrics inherits exactly this blind spot. Silent data corruption is also the failure class that status-code and latency dashboards are least equipped to catch: every signal looks healthy while the underlying numbers are wrong. The practical takeaways for any team running progressive delivery are to make canaries participate in all traffic paths, including consumer groups, and to pair operational metrics with output verification such as shadow replays and business-level diffs.