Teams that added an AI reviewer are all trying to answer one question: how much review work is actually left for humans? The number they reach for is findings per PR, or findings per thousand lines of diff. I went to compute it over a window of merged pull requests and got it wrong three times before it held still. Each wrong version is a different way the measurement fails, and all three reproduce in about ten minutes.
Everything below is from merged PRs in microsoft/vscode, pulled through the GitHub REST API on a single afternoon. The sample is the most-recently-updated page of closed PRs (sort=updated&direction=desc), filtered to merged, which is a recently-touched slice rather than a random sample. I say that up front because it matters: a slice sorted by update time is not a population, and any latency number you read off it is a tail estimate.
Wrong answer one: zero, and zero was an artifact
The first version counted top-level comments per PR. On the list endpoint, comments and review_comments come back as null for every entry. They are only populated by the single-PR endpoint, which is one request per PR. Build a dashboard on the list response and you get a column of nulls that renders as "no findings."
The second version filtered inline comments by login.endswith('[bot]') and also got zero. Copilot's inline review comments are authored by the login Copilot, no bot suffix anywhere, and each one carries a pull_request_review_id that points back to the review object it belongs to. The review object itself is authored by copilot-pull-request-reviewer[bot] and is always in state COMMENTED. So one review has two identities in the API and a state that never means approved. Filter on either identity alone and you mis-count. The join is the only thing that works:
rev = get(f"/repos/{repo}/pulls/{n}/reviews")
cmts = get(f"/repos/{repo}/pulls/{n}/comments?per_page=100")
bot = [r for r in rev if r["user"]["login"] == "copilot-pull-request-reviewer[bot]"]
latest = bot[-1]
inline = [c for c in cmts
if c["user"]["login"] == "Copilot"
and c["pull_request_review_id"] == latest["id"]]
Two identities, one review. That is the whole trap.
Wrong answer two: the verdict body changed shape four times in ten weeks
I wrote a regex for the body format, ran it, and half the sample came back unparsed. Pulling the raw bodies explained it. The same tool shipped four different body shapes inside the window I sampled:
| Shape | Marker in the body | Submitted dates in sample |
|---|---|---|
findings_field |
**Findings:** 4 and **Review effort:** Balanced
|
2026-09-04 to 2026-09-24 |
open_issue_summary |
<strong>1 open issue</strong> inside <details open>
|
2026-10-16 to 2026-11-06 |
no_open_issues |
**No open issues** |
2026-11-08 |
| legacy / unparsed |
**Open (4)** with a severity icon and no findings count |
2026-09-21 |
A parser keyed on **Findings:** reads every October and November review as malformed. A parser keyed on open issue misses September entirely. And because the count line only exists when findings exist, a clean verdict drops the field completely, so the naive read of a clean review is "unparsed," which most pipelines then count as zero.
The verdict strings moved too. Same tool, same repository, four headings: 🟡 Changes recommended, 🟢 Approval recommended, 🔵 Needs a closer look, and older bodies with no heading at all. The emoji prefix means an anchor or startswith check on the heading fails silently.
Here is the parser that finally held. Order matters, and unparsed is a real outcome, never a zero:
def parse(body):
verdict = None
for needle, label in [("Approval recommended", "approval"),
("Changes recommended", "changes"),
("Needs a closer look", "closer")]:
if needle in body:
verdict = label
break
if "No open issues" in body:
return verdict, 0, "no_open_issues"
m = re.search(r"\*\*Findings:\*\*\s*(None|\d+)", body)
if m:
return verdict, 0 if m.group(1) == "None" else int(m.group(1)), "findings_field"
m = re.search(r"(\d+)\s+open\s+(?:issues|issue|finding)", body)
if m:
return verdict, int(m.group(1)), "open_issue_summary"
m = re.search(r"<summary><strong>Open \((\d+)\)</strong>", body)
if m:
return verdict, int(m.group(1)), "open_paren"
return verdict, None, "unparsed"
If your findings counter has been reporting a stable number all quarter, check whether it is actually parsing, or whether it is silently reporting zero for a body shape that shipped in October. The REST API docs for pull request reviews describe the review object and its states, but they do not describe what any given reviewer puts in the body, because that is the vendor's business. That is the part you have to re-verify yourself.
Wrong answer three: the reviewed commit wasn't the code that merged
I assumed each review was about the code that landed. It wasn't. The review object carries a commit_id, and the PR carries head.sha. Comparing them:
- 25 of 28 merged PRs had a Copilot review whose
commit_idwas not the PR head. - 3 were on-head.
- 2 of the on-head reviews were approvals.
The off-head gap is not a rounding error, it is the normal case. PRs get rebased, amended, and force-pushed after the bot comments, so the review is a statement about a revision that no longer exists. One wrinkle worth knowing if you try to measure the gap in commits: after a rebase the reviewed commit may not appear in the PR's commit list at all, even though it still exists in the repository. Count commits-since-review from that list and you get nonsense. Compare the two SHAs.
Why nobody catches it: GitHub's dismiss-stale-approvals setting, described in the protected branches docs, applies to reviews in state APPROVED. Bot reviews are posted as COMMENTED, and a COMMENTED review never expires. So the one mechanism designed to invalidate a review of stale code does not apply to the reviews these tools produce.
This is why I keep coming back to the input side of the problem rather than the finding count. My earlier post on checking the input distribution of AI code review for large PRs covers the sizing half, and the PR wait time numbers after adding an AI reviewer cover the queue half. Neither number means anything if the verdict you are counting was written about a different revision.
What the counter reports once it parses
With all three fixes in, across 28 merged PRs ranging from 9 lines to 28,674 lines of churn (2 to 317 files, 4 to 172 commits):
- Median findings: 1.
- Six PRs with zero findings, all with a verdict of
Changes recommended,Needs a closer look, orApproval recommended. - Verdict spread: 20
Changes recommended, 4Needs a closer look, 3Approval recommended. - One merged PR had no Copilot review object at all: 7,005 lines, 54 files, 74 commits, merged anyway.
The largest PR in the sample, 28,674 lines across 317 files, came back with one finding. The smallest, 9 lines across 2 files, came back with zero. Findings do not scale with diff size, but the bill does, which is what I found when I looked at the 10,078-line PR with zero findings. The density number is only meaningful next to the coverage number and the off-head number.
It isn't only Copilot
I ran the same comparison on another reviewer in another repository. On supabase/supabase, one bot produced 41 review objects across 14 merged PRs, with 13 of those objects on-head and 28 off-head, all in state COMMENTED. Same pattern, different vendor.
That 41-over-14 ratio is its own measurement decision. If your metric is findings per review object, one chatty reviewer looks twice as productive as a quiet one that covers the same PRs. If your metric is findings per PR, you have to pick one object per PR and say which. I take the last object per reviewer per PR, and I report the object count so the choice is visible.
The counter worth building
- Take the findings count from the review object body, and use inline comments as a cross-check joined by
pull_request_review_id. Do not use either one alone. - Parse every known body shape. Emit
unparsedas its own value. An unparsed verdict is a bug in your pipeline, not a clean review. - Compute off-head rate as a first-class metric, comparing
commit_idtohead.sha. Report it next to findings, not buried in a log. - Report coverage: merged PRs with no review object at all, over the same denominator. In this sample that was 1 in 28, on a PR with 74 commits.
- Pin the window and take the denominator from the same window.
merged:date qualifiers in the search API do not behave likecreated:ones, so build the cohort oncreated:plusis:mergedand keep it fixed. - Print the counter's own error rate. Mine was 100% wrong three times before it was right once.
The reproducible protocol for measuring PR review time with AI applies the same idea to the latency side. Both come down to the same rule: your measurement instrument needs its own test, or you are reporting your parser's bugs as the tool's behaviour.
What this changes about the review-volume question
The question "how do we review the growing volume of AI-generated code" usually gets answered with a findings-per-PR chart. That chart is three measurements stacked on top of each other, and any one of them being wrong makes the whole thing decoration: coverage (did the bot review this PR at all), freshness (was the verdict about the code that merged), and density (how many findings, per what denominator).
When you evaluate a reviewer, ask what its API exposes before you ask how good its findings are. A tool that posts a structured verdict with a severity and a count in a stable place is measurable in an afternoon. A tool that only posts free-text comments forces you to build a parser that breaks every time the wording moves, which is exactly what happened here across ten weeks. That question is tool-agnostic, and it applies to CodeRabbit and Greptile the same way it applies to Copilot. If self-hosting is on your list, options like Kodus put the review in your own infrastructure, which matters mostly because you can then instrument it directly instead of scraping a vendor's markdown.
The other half of the answer is the one that keeps showing up in the data: an AI reviewer judging output produced by an AI agent is one model's opinion measured repeatedly, and that blind spot does not shrink with volume. Cross-model review plus something that is not a model at all (types, tests, a compile step) is what gives the human reviewer a queue they can trust. The counter tells you how big that queue is. It cannot tell you the queue is right, and if it is misreading the verdicts it cannot even tell you how big it is.
Top comments (0)