Audit of 55 merged VS Code PRs finds Copilot code review never blocks a merge

Copilot reviews always land in the COMMENTED state

An audit of 55 merged pull requests from microsoft/vscode, published on dev.to by Cole Halton, found that GitHub Copilot code review set the same review state every single time: COMMENTED. The sample came from two slices of recently created PRs filtered to merged ones, with 28 Copilot reviews in the first slice and 27 in the second. Across all 55 there were zero APPROVED states and zero CHANGES_REQUESTED states.

The distinction matters because GitHub's REST API models a review as a group of comments plus a state, and branch protection reads that state. A COMMENTED review neither satisfies required approvals nor triggers the merge block that follows when changes have been requested. The optional setting that dismisses approvals after the diff changes is equally irrelevant, since a COMMENTED review never records an approval to dismiss.

The verdict heading is UI text, not state

Headings such as "Approval recommended" and "Changes recommended" are presentation, not state. According to the dev.to post, GitHub's own documentation says Copilot reviews do not count toward required approvals by default, and enabling Copilot approvals is a separate setting at repository, organization and enterprise level.

Despite that, roughly half of the merged PRs carried a non-approval verdict at merge time: 12 of 28 in the first slice and 15 of 27 in the second. The first slice split into 16 "Approval recommended", 9 "Changes recommended" and 3 "Needs a closer look".

The heading and the findings list disagree

The audit found the verdict heading and the structured findings list are computed separately. In the first slice, 19 of 28 final reviews reported zero open findings, yet three of those still carried "Needs a closer look".

The clearest example is PR 337469 in microsoft/vscode, which received two Copilot reviews 52 minutes apart on the same commit. The first described four unresolved moderate issues, the second said three remained, and both bodies ended with the structured line "Findings: None". The second review also flagged a "previously missed issue" in code that had not changed since the first pass.

Most verdicts describe a diff that no longer exists

Comparing each review's commit_id against the PR head at merge time, the audit found 13 of 28 final reviews in the first slice and 15 of 27 in the second were rendered against a commit that was not the head when the PR merged. Because a COMMENTED review records no approved diff state and nothing refreshes it, an off-head verdict is the normal end state rather than an exception.

Re-reviews are rare and report a delta

In an updated-sorted slice of 28 merged PRs, 24 had exactly one Copilot review; across the two created-sorted slices, only 3 of 55 had more than one. A changelog entry dated November 6 says that on later reviews of the same PR, findings unrelated to recent changes are collapsed and then hidden, so a second review is a delta against the last push rather than a fresh read of the whole change. A drop in findings between two runs can therefore reflect narrower scope, not genuine improvement.

What to record if you measure Copilot reviews

The author recommends five fields per review run: the reviewed commit id and the PR head sha at read time, a findings count parsed from the structured body, the effort level in use, the verdict heading stored as text with its read date, and whether a second review ran and what changed between runs.

Effort level matters because the tiers behave like different products: documented estimates put average credit consumption around $0.25 per review for Lite, $1.50 for Balanced and $10 for Max, and Balanced now uses three reviewer models plus a fourth that combines their findings. The same November update moved the effort label into a compact footer, consolidated skipped and excluded files into one "Files not reviewed" section, and added attribution fields to the REST and GraphQL APIs. Metrics should key on those fields, the post argues, rather than grepping heading strings that break on a UI release.

If you need a gate, buy a gate

The post contrasts Copilot with tools whose gating behavior is documented. CodeRabbit exposes an auto-approvals flag, off by default, and states that missing, failed, disabled or stale required assessments withhold approval. Kodus is self-hosted under AGPLv3 with bring-your-own-model and review rules scoped to an organization, repository or path. In both cases you can predict from the documentation what happens to the verdict when the diff changes; with Copilot's default configuration, nothing happens.

Why it matters

The author is explicit that this is not a bug report: advisory review is a defensible design, and teams treating Copilot as advisory get what the product promises. The risk sits in the in-between cases. Anyone counting "AI approvals" or "AI blocks" from review state will count zero forever, on every repository; anyone counting from heading text is measuring a string GitHub rewrites on its own schedule. And any team quietly assuming the AI reviewer holds the line before merge should know the verdict never gates anything, often describes a stale diff, and can change its answer when re-run on identical code.