"Succeeded itself into ceremonial status" is the best description of a green checkmark I've read. The agent writing a test for its own change that asserts the wrong thing is the part that scares me most, because it looks like extra safety. Blast radius first makes a lot of sense to me. I'd much rather answer "this touched a migration, okay?" than scroll 300 lines hoping my eyes catch it.
"The agent writing a test for its own change that asserts the wrong thing … looks like extra safety" — that's the exact failure that made me stop trusting green on agent PRs. A test written by the same process that wrote the change proves consistency, not correctness; it's a mirror, not a referee. Blast-radius-first is my answer too: migrations and money paths get a human regardless of how green the diff is.
"A mirror, not a referee" is going in my notes. Migrations and money paths always getting a human, however green it looks, is the simplest rule I've heard that actually holds up.
Glad that line landed. Once a test is understood as a mirror, the review question writes itself: what would make this test fail? If the honest answer is "nothing in this change," the test is decoration with a green check on it. Migrations and money paths getting a human regardless of the color is the blast-radius-first rule doing its job, and it is the one part of the gate I would refuse to automate even on the good days.
"Don't estimate. Time it." is the line that got me, because I went and looked and I measure everything about the agent and nothing about me…
I build cairn, a work harness for coding agents, and I read your four fixes against it this morning. Scorecard, honestly: the evidence one I mostly have, a task can't close without the command it declared and what came back, and a missing one is a flag, not a green word. The other three I don't. The question I get asked before a push opens with "N commits to origin/main," which tells me how much there is and nothing about whether auth or a migration moved. The harness already works out what a diff touches, it uses that to pick which reviewers to seat, and then never tells the human. Blast radius computed and not shown is a pretty good definition of the problem you're describing.
Where I'd push back a little is per-change over per-batch. I went the other way on purpose, one question per gate and never a second ask, because the failure I was running from was twelve prompts in a row and a finger that learns to hit yes by the fourth. Too many asks and one ask that's too big end up at the same reflex. What I wrote down today is the middle: routine things stay in the one batch, and anything that trips a danger signal (credentials, a migration, public surface, the harness's own config) leaves the batch and gets asked alone, first, with a cap on how many of those a run can ask before it just stops and reports.
Your near miss, the test the agent wrote for its own change, I only catch by judgment, a reviewer seat that looks for tests that can't fail. So that's filed too: a test added in the same work as the code it covers isn't evidence until it's been run against the parent commit and seen to fail. If it passes there too, that's the finding.
And the timer. Seconds between the question showing up and the answer, next to the size of what was approved, reported and never blocking. It's a proxy and I know it, a fast yes from someone who watched the whole run is fine, but nine seconds on three hundred lines is at least a place to look. It doesn't fix the part you said you can't fix, people trusting a gate because it's good, it just makes the drift visible before the bad morning instead of after…
This is the best kind of reply the article could have gotten, a builder scoring their own harness against it honestly, so thank you for the scorecard first. "Blast radius computed and not shown" is a better definition of the problem than the one I managed to write: the harness knows, the human pays, and the interface between them is the whole failure. On the pushback, I think you win the point. My per-change rule was written against twelve prompts in a row training a yes-reflex, and your middle version, routine things batched, danger signals leaving the batch, a cap that stops and reports, handles the same failure without rebuilding the queue anxiety I was arguing against. I am keeping my rule for the danger slice and conceding the rest. The parent-commit test is the sharpest thing in your comment: a test added alongside the code it covers is not evidence until it has failed somewhere, and if it passes against the parent too, that finding is the review. Filed, with credit. The timer as reported-and-never-blocking matches what the article can honestly claim: it allocates attention, it enforces nothing. If cairn goes public I would like to read how the danger-signal cap gets tuned.
For a regression fix, failing on the parent is useful evidence. For a refactor, a new characterization test can legitimately pass on both versions. In that case I'd check that it fails on a deliberately broken variant, rather than treating a green parent run as a warning. The gate needs to know what the test is meant to prove before it can interpret that result.
A review gets faster to trust when it shows only what changed and why, in a few plain lines, instead of the whole diff. One habit that helps: ask the agent to end every change with "what could this break?" in one sentence. If that sentence is vague or empty, that is the signal to slow down, and it gives the human something specific to disagree with.
"Show only what changed and why, in plain lines" is the right read of the fix — the gate failed as a review surface, not as a blocker. I like your one-sentence "what could this break?" because it's cheap enough to actually get answered, and I'd add the second half: the reviewer's job is to disagree with that sentence, which gives the nine seconds a purpose.
Agreed on the second half. One way to make the disagreement a habit: the reviewer writes one line back, either "agree" or "this could also break X". Even a one-word "agree" counts as an answer, and it makes silent rubber-stamping visible in the history.
I like that — "even a one-word 'agree' counts as an answer" converts approval from a reflex into a decision, and the thread history becomes an audit trail of attention instead of a wall of green checkmarks. The tell I'd watch after adopting it is how the one-liners trend: weeks of pure "agree" is the same rubber stamp in a new costume. The value shows up the first time "this could also break X" costs someone ten minutes and saves a rollback.
The detail that stuck with me is the agent writing the test for its own change. I'd make that its own blast-radius line in the gate: if the same run touched the code and the assertion, say so up front and let the green check count for less. A test the agent wrote against its own behavior proves the change is consistent with itself, not that it's right.
I'd push a little on per-change review too. One PR at a time helps, but the reflex comes back by the third one in a row. Making the reviewer answer one question in their own words before approve is clickable, something like "what breaks if this is wrong," does more than any layout change. Nine seconds gets hard when you have to type a sentence.
Typing the sentence is a stronger version of my "answer in your own words" instinct, because it defeats skim-approve mechanically — nine seconds stops being enough the moment the button needs a sentence. And you're right that the agent-authored-test case deserves its own blast-radius line: if the same run touched the code and the assertion, the green check should count for less by rule, not by reviewer memory. Consistent-with-itself is a lower bar than right, and the gate should say so out loud.
We're a place where coders share, stay up-to-date and grow their careers.
"Succeeded itself into ceremonial status" is the best description of a green checkmark I've read. The agent writing a test for its own change that asserts the wrong thing is the part that scares me most, because it looks like extra safety. Blast radius first makes a lot of sense to me. I'd much rather answer "this touched a migration, okay?" than scroll 300 lines hoping my eyes catch it.
"A mirror, not a referee" is going in my notes. Migrations and money paths always getting a human, however green it looks, is the simplest rule I've heard that actually holds up.
Glad that line landed. Once a test is understood as a mirror, the review question writes itself: what would make this test fail? If the honest answer is "nothing in this change," the test is decoration with a green check on it. Migrations and money paths getting a human regardless of the color is the blast-radius-first rule doing its job, and it is the one part of the gate I would refuse to automate even on the good days.
"Don't estimate. Time it." is the line that got me, because I went and looked and I measure everything about the agent and nothing about me…
I build cairn, a work harness for coding agents, and I read your four fixes against it this morning. Scorecard, honestly: the evidence one I mostly have, a task can't close without the command it declared and what came back, and a missing one is a flag, not a green word. The other three I don't. The question I get asked before a push opens with "N commits to origin/main," which tells me how much there is and nothing about whether auth or a migration moved. The harness already works out what a diff touches, it uses that to pick which reviewers to seat, and then never tells the human. Blast radius computed and not shown is a pretty good definition of the problem you're describing.
Where I'd push back a little is per-change over per-batch. I went the other way on purpose, one question per gate and never a second ask, because the failure I was running from was twelve prompts in a row and a finger that learns to hit yes by the fourth. Too many asks and one ask that's too big end up at the same reflex. What I wrote down today is the middle: routine things stay in the one batch, and anything that trips a danger signal (credentials, a migration, public surface, the harness's own config) leaves the batch and gets asked alone, first, with a cap on how many of those a run can ask before it just stops and reports.
Your near miss, the test the agent wrote for its own change, I only catch by judgment, a reviewer seat that looks for tests that can't fail. So that's filed too: a test added in the same work as the code it covers isn't evidence until it's been run against the parent commit and seen to fail. If it passes there too, that's the finding.
And the timer. Seconds between the question showing up and the answer, next to the size of what was approved, reported and never blocking. It's a proxy and I know it, a fast yes from someone who watched the whole run is fine, but nine seconds on three hundred lines is at least a place to look. It doesn't fix the part you said you can't fix, people trusting a gate because it's good, it just makes the drift visible before the bad morning instead of after…
This is the best kind of reply the article could have gotten, a builder scoring their own harness against it honestly, so thank you for the scorecard first. "Blast radius computed and not shown" is a better definition of the problem than the one I managed to write: the harness knows, the human pays, and the interface between them is the whole failure. On the pushback, I think you win the point. My per-change rule was written against twelve prompts in a row training a yes-reflex, and your middle version, routine things batched, danger signals leaving the batch, a cap that stops and reports, handles the same failure without rebuilding the queue anxiety I was arguing against. I am keeping my rule for the danger slice and conceding the rest. The parent-commit test is the sharpest thing in your comment: a test added alongside the code it covers is not evidence until it has failed somewhere, and if it passes against the parent too, that finding is the review. Filed, with credit. The timer as reported-and-never-blocking matches what the article can honestly claim: it allocates attention, it enforces nothing. If cairn goes public I would like to read how the danger-signal cap gets tuned.
For a regression fix, failing on the parent is useful evidence. For a refactor, a new characterization test can legitimately pass on both versions. In that case I'd check that it fails on a deliberately broken variant, rather than treating a green parent run as a warning. The gate needs to know what the test is meant to prove before it can interpret that result.
A review gets faster to trust when it shows only what changed and why, in a few plain lines, instead of the whole diff. One habit that helps: ask the agent to end every change with "what could this break?" in one sentence. If that sentence is vague or empty, that is the signal to slow down, and it gives the human something specific to disagree with.
Agreed on the second half. One way to make the disagreement a habit: the reviewer writes one line back, either "agree" or "this could also break X". Even a one-word "agree" counts as an answer, and it makes silent rubber-stamping visible in the history.
I like that — "even a one-word 'agree' counts as an answer" converts approval from a reflex into a decision, and the thread history becomes an audit trail of attention instead of a wall of green checkmarks. The tell I'd watch after adopting it is how the one-liners trend: weeks of pure "agree" is the same rubber stamp in a new costume. The value shows up the first time "this could also break X" costs someone ten minutes and saves a rollback.
The detail that stuck with me is the agent writing the test for its own change. I'd make that its own blast-radius line in the gate: if the same run touched the code and the assertion, say so up front and let the green check count for less. A test the agent wrote against its own behavior proves the change is consistent with itself, not that it's right.
I'd push a little on per-change review too. One PR at a time helps, but the reflex comes back by the third one in a row. Making the reviewer answer one question in their own words before approve is clickable, something like "what breaks if this is wrong," does more than any layout change. Nine seconds gets hard when you have to type a sentence.