The two replies that have cost me the most time are short ones. "It's fixed." And "that isn't used anywhere." Both sound like a place you can stop. I would read one, move on to the next thing, and only find out later that nothing had run since the last edit, or that the agent had searched once and decided the question was settled.
I run a one-person company on long-running Claude Code sessions and scheduled AI jobs. When an agent tells me a bug is gone and I believe it, I move on. The mistake usually shows up after I've already started something else.
So I wrote a Stop hook. When the agent tries to end its turn after a claim like "fixed," "tests pass," "deployed," or "doesn't exist," the hook looks at what actually happened in the session. If the proof isn't there, it sends the agent back once and says what's missing. The note looks like this:
claim-check: you wrote "is fixed", but nothing ran successfully after the last edit.
Run the check that proves it (the tests, the script, a request to the live URL) and
show the result, or say plainly that it isn't verified yet.
It never blocks twice in a row, so the agent can't get stuck. If the hook itself crashes, it lets the turn end. And if the agent just says the work isn't verified yet, that always passes. I wanted a way to make it show its work, not a way to trap it.
On November 4, 2026, I ran the hook over my own Claude Code history to see what it would have done. After dropping 1,649 duplicate turns (resuming a session copies earlier turns into a new file), I had 3,977 turns left. 406 of them ended with a claim like "done," "fixed," "deployed," or "tests pass."
The version I measured this way, 0.2.0, would have blocked 86 of those 406. Most of the 86 were not the case I had built it for.
77 had no tool calls at all. Those were usually wrap-ups: the agent saying something was fixed or deployed, with no commit, test result, or status code in the reply. The proof may have been in an earlier turn, but the wrap-up is what I actually read, and it didn't show any. 8 had tool calls, but only reads, searches, read-only commands, to-do updates, or failed commands. 1 made the claim after a code edit with nothing run successfully since. That last one is the case I cared about. It's probably rare in my numbers because an older version of the hook was already catching it on my machine. Your count may be higher.
I hand-checked 24 of the 86: 20 drawn at random when the count was still 82, plus 4 that later rule changes added. Eighteen were wrap-ups that claimed finished work without citing a commit, test output, or status code. Three weren't claims of finished work at all: a research finding (a change had been tested and made the model worse), a condition (the work would count as done once merged), and a plan for work that hadn't started. The other three could be read either way.
After I posted those numbers, someone argued that the 77 wrap-ups were the ones that mattered, because a hook that blocks a correct wrap-up gets turned off. They had a point. In 76 of those 77, something had run successfully after the last edit, just in an earlier turn. So 0.3.0 started counting a check from earlier in the session, as long as no code was edited after it. Of the same 406 claims, it now blocks 2: the one after an edit with nothing run, and one wrap-up with nothing run since the last edit.
The tradeoff is real. A wrap-up no longer has to show its proof, so the wrap-ups I hand-checked get through now too. I'd rather catch the dangerous case and stay installed than win the argument about tidy endings.
Separately, 17 turns ended by saying something wasn't there, and in 2 of them the agent had looked once or not at all. That's the other reply that has burned me. For a "doesn't exist" or "nothing calls it" claim, the hook wants two lookups in that turn, of different kinds. Searching for the same text twice doesn't count, even in another folder or with different flags. A different text, a different file, or a different tool does.
For a "fixed" or "tests pass" claim, what it wants is simpler. Something has to have run after the last code edit and not failed: a shell command, an MCP tool, a URL fetch, or a subagent. git diff and cat don't count, and neither does a command that failed. If the turn didn't edit code, a successful check from earlier in the session is enough, as long as no code was edited after it. The hook will also accept proof cited in the reply itself: a commit hash, a line like "14 passed" with no failures, a 2xx status, "exit code 0," or a link to a CI run. "3 failed, 2 passed" does not count, and neither does "0 passed."
A few of the gaps are worth saying out loud, because they are how you end up turning a hook off.
It matches set phrases. "The redirect loop is fixed" and "Fixed." get caught. "Fixed the redirect loop," "Fixed, deployed to prod," and "I'm done" do not. A sentence that opens with a condition, like "After the restart, everything is deployed," gets through. One honest "not verified" anywhere lets the whole reply through, including an unrelated one like "Fixed. Do not test on prod." It only understands claims in English and Chinese. And a command after the edit only proves that something ran. It doesn't prove the command tested the right thing.
If you want to see what it would do on your own history before it starts sending the agent back, clone the repo and run python3 claim_check.py replay. It reads past sessions in ~/.claude/projects and prints how many claims it would have blocked. Nothing leaves your machine. You can also set the plugin's log_only option to true, leave it for a week, and read the log before you turn blocking on.
For Claude Code, the install is:
/plugin marketplace add Balance0014/claude-code-claim-check
/plugin install claim-check@claim-check
Then reload plugins if Claude Code asks, or start a new session. It needs Python 3.9+ and has no dependencies. If you don't use the plugin system, you can point a Stop hook in ~/.claude/settings.json at claim_check.py. For Codex and other agents, npx skills add Balance0014/claude-code-claim-check installs the rules as a skill. The agent can read them, but nothing makes it follow them. There's no hook for Codex in this release.
The hook and the replay script are in claim-check. In the week before I published the first version, the older copy on my machine blocked 19 times. That was enough to keep it on.
If the thing eating your week isn't a false "fixed" but the way long conversations get re-read on every reply, I also made when-to-clear. It reads the token counts Claude Code already saves locally, tells you when it's time to /clear, and can write a handoff note so a new session can pick up the work. Different problem, same machine.
Top comments (39)
For the exit-0 gap, the check that has worked best for me costs one extra run: take the edit back out, run the same command, and it has to fail. If it still passes without the fix, it never exercised the fix, whatever the exit code said. Then put the edit back. It doesn't need the transcript at all, and the agent can do it itself before it says "fixed".
The claim that bit me yesterday was "released". The PR was green, the merge was green, and the publish job that runs after the merge had failed on an unrelated flaky test, so the registry still served the previous version. A link to the CI run would have passed your hook, and for that claim it proves the wrong job. What settled it was asking the registry which version it serves. For "deployed", the proof lives at the destination, not in the pipeline.
Thanks, both of these are better than what the hook does today. The "take the edit back out and it has to fail" check is the one I keep wishing I could ask the agent to do by default. My worry is slow test suites, where a second run costs real time, so I'd probably only ask for it when the claim is about a bug fix.
The "released" example is a good catch. You're right that a green CI link proves the wrong job there. The hook only checks that something ran, not that it looked at the destination, so asking the registry which version it serves is a gap I can't close with a transcript check alone.
That distinction will pay off: a generic missing-proof note tells the agent to work harder; a typed requirement tells it how to make the claim falsifiable. I’d let the evidence contract name the scope too—test target and environment, deployment artifact and environment, search roots and exclusions—so proof for one scope cannot be recycled for another. It keeps the feedback actionable without turning the hook into a heavyweight policy engine.
Yes. A typed evidence form still fails if the proof is from the wrong scope. A green test on a different target, or a staging deployment id offered as production proof, would pass a contract that only names the shape of the evidence.
I'll keep those fields small when I tighten the contract: test target and environment, the deployment artifact and its environment, and for unused-code claims the search roots plus exclusions. That stops proof from sliding across scopes, and it still leaves policy out of the hook.
On slow suites, the second run only needs one test: the one that's supposed to prove the fix. Take the edit out, run that test, put the edit back. That's usually seconds, not a second full run. Keeping it to bug-fix claims sounds right to me.
For "released", maybe the hook doesn't need to ask the registry itself, only to see the registry's answer. If the claim is "released" and the transcript has no command whose output names the new version, it sends the agent back, the same way it does for "fixed" without a run. It stays a transcript check, with the destination as the evidence it looks for.
That's a much cheaper second run than what I was picturing, thanks. One test, out, back in, usually seconds. I'll try that.
Your "released" idea keeps it a transcript check, which I like. The failure mode I'd still watch for is a command that prints the version from a local file, like package.json or a git tag, instead of from the registry. The output names the new version, but it never asked the destination. That's the same kind of miss as a green CI link proving the wrong job. I'd want the transcript to show it actually queried the registry, not just that the version string showed up.
The local file is the easy way to pass that check without asking anything. It's also what nearly fooled us: the version in the repo was already bumped, and only asking the registry showed the publish hadn't landed yet.
So I'd match the command, not just the string. The transcript needs a call that names the destination, like
npm view <pkg> version,pip index versions <pkg>or a request to the registry URL, and it has to come after the publish step. A version that shows up any other way I wouldn't count.The set-phrase gap is the one I'd worry about, because an agent can say "done" a hundred ways. In our pipeline at Oizom we gate the action instead of the words: the agent ends its work by calling endWork, and that call is refused until its plan is finished, its stack is healthy and the merge is clean. Your replay over 406 real claims is the part I'd copy. Did the 77 wrap-ups cluster at the end of long sessions, or spread evenly?
I didn't split the 77 by session length, so I can't say yet. I only tagged them as wrap-ups where the check had run in an earlier turn. That's a useful next cut, especially to see whether longer sessions are where the stale "done" signal shows up.
Fair, that cut can wait. From the same replay I'd want to know about the 2 that v0.3.0 still blocks: were they real cases where nothing ran, or false alarms on a phrase? Two out of 406 is small enough to read by hand.
They weren't false alarms on a phrase. Of those two, one is the case I built the hook for: a claim right after an edit, with nothing run. The other is a wrap-up that also had nothing run since the last edit. So both are real "nothing ran" cases, just at different points in the session.
Then v0.3.0 blocks nothing it shouldn't on that replay, which is what keeps a hook like this switched on. The second case is the sneakier one: a check that ran before the last edit still looks like proof. Splitting the 77 by session length should show how often that happens.
Yeah, that split is worth doing. The 77 were cases where a check had run earlier in the turn, but I haven't grouped them by session length yet. I also want to compare that with time since the last edit, since a long session isn't automatically stale if nothing changed after the test.
The rule that an honest "not verified yet" always passes is the part I like most. I came to something similar from the other side: in my setup a check that couldn't give an answer, like a timeout or a diff it couldn't read, is a separate state, "don't know", not a pass and not a fail. Before that, "done", "not checked" and "the process just died" all showed the same green badge. About the gap you're least happy with, exit 0 not proving the command touched the edited code: I don't think the transcript can answer that either. What helped me was moving the question one level up. A separate read-only agent checks each phase against the plan, and the tests are written from the plan, not by the agent after its own code. It doesn't prove coverage, but at least someone asks whether the right thing was tested.
That distinction between “don’t know” and pass or fail is useful, especially when a timeout or unreadable diff makes the evidence incomplete. My Stop hook is deliberately narrower: it catches a confident “it’s fixed” claim when I can’t find proof in the session, but an exit code still can’t tell me whether the right code was tested. I like the read-only plan check as a separate layer rather than making the hook pretend it can prove coverage.
Agreed, keeping them as separate layers is cleaner. A hook that catches a confident "it's fixed" without proof is already a big win, and it doesn't need to prove more than it can. The exit code problem is real though: green tests only tell you something ran, not that it touched the code that changed. That's the gap I still don't have a good answer for.
The check that's caught this for me is cheaper than trying to prove coverage. After a green run, I look at whether the test process actually imported or opened the file I just changed. If that path never shows up, the green result is about old behavior, not the edit.
It won't catch a weak assertion. For a bug-fix claim, the stronger version is: take the edit back out, run the same test, and it has to fail, then put the edit back. That still doesn't need to live in the hook. The hook can keep doing what it's good at, and this stays a separate check when the claim is that a specific bug is gone.
Accepting "a shell command, an MCP tool, a URL fetch, or a subagent" as the thing that ran leaves one path unchecked. Per the hooks docs, Stop runs when the main agent finishes responding, and a subagent finishing fires SubagentStop instead. The plugin's hooks.json registers only Stop. So if the main agent delegates the test run and the subagent reports "all tests pass" without running them, nothing checks that reply, and the Agent call then counts as the successful run for the main turn's claim.
SubagentStop looks like a fit for the same script. Its input carries
last_assistant_messagewith the subagent's final text, it accepts the same block decision as Stop, and the block reason is delivered to the subagent as its next instruction. Two details in that section need handling.transcript_pathis still the main session's transcript there, and the subagent's own file is inagent_transcript_path. The event also fires for internal agents such as prompt suggestions, whereagent_typeis the session's agent name or an empty string, and a matcher that names agent types skips the empty ones.If Stop only runs when the main agent finishes, then a delegated test run would sneak through exactly the way you described. The Agent call itself can look like a successful run, and the subagent's "all tests pass" never gets checked.
I'll look at adding SubagentStop with the same script, and I'll be careful about the transcript_path vs agent_transcript_path distinction you flagged. The empty agent_type case is easy to miss too; a matcher that only lists named types would skip exactly the internal ones I wouldn't want to ignore, or worse, trip on them. I'll need to handle that explicitly rather than assuming a type list is enough.
Thanks for laying this out so clearly. This is the kind of hole that would have stayed invisible in my own replay set, because most of those turns weren't delegated.
@gapmine Agreed, a count only means something next to the count it should be. Keeping the expected number in the repo, so a drop shows up in the diff, would catch the short-run case without anyone reading a log.
Yeah, putting the expected count in the repo is cleaner than keeping it in the hook. The catch I'd watch is if that file is something the agent can edit. A short run can then pass by lowering the number in the same turn, and the diff still looks deliberate. I'd want CI to own that file, or at least flag it when it changes in the same commit as a test run.
The drop from 86 blocks down to 2 out of 406 claims explains why most verification controls get turned off. In v0.2.0, roughly 98% of the blocks interrupted valid wrap-ups where a check had already run earlier in the session, which creates enough friction that people uninstall the hook inside a week.
Accepting an earlier-turn check when no code changed afterward trades tidy wrap-up formatting for keeping the guardrail enabled. The remaining gap you pointed out is that an exit code of zero only shows a command finished, leaving open whether it exercised the edited branch. Trying to prove full test coverage from transcript replay hits diminishing returns fast, so blocking the silent no-run claim right after an edit captures most of the downside without bogging down normal turns.
Yeah, that remaining gap is the one I'm least happy about. Exit 0 only means a command finished, not that it touched the branch I just edited. I poked at inferring that from the transcript and it fell apart pretty quickly, too many ways a test command can look.
So I stuck with blocking the silent no-run right after an edit. That's the case that actually wasted my time, and it's cheap enough that I leave the hook on. A later check that the command even mentioned the edited files might be worth it, but I'd still leave real coverage to CI.
zerokool1986's "marked disproven, reason attached" point is where receipt design gets hard. Keeping the wrong verdict visible solves the re-derivation problem, but it leaves the graph question open: a disproof that exists but nothing links to gets rediscovered anyway.
I'd go one step further and make the disproof a first-class record that points both ways. The original verdict stays untouched, a new disproof entry references it, and the receipt keeps a list of disproof links that renders next to the claim. Then "it's fixed" becomes a state entry that carries its own overturn history instead of a sentence the next session has to re-litigate.
The part I never get right is the empty-state shape. A list that starts empty tends to stay empty; a count of disproofs reads like a scoreboard against the agent. What does the slot look like on day one, before any disproof exists?
Thanks, I like the concern about the empty state. I'd render it as plain status text, not a zero-count badge: “No later correction linked yet.” That says only that the graph is empty, not that the original claim was verified. Once there is a correction, show the linked record next to the original verdict.
@gapmine agreed, and that is the gap to close. A 'fixed' that arrives with the command and the last lines it printed can be checked; without the tail, '0 tests collected' passes the same way a real run does. Requiring that output tail in the claim closes exactly that hole.
Agreed. A clean exit can still hide a run that collected zero tests. I'd want the receipt to include the test count and the last output too, not just the command and exit code. That's much harder to mistake for a real pass.
The Stop hook changes the completion claim from prose into something the session can substantiate. I’d bind each claim type to minimal evidence: command plus exit status for tests, immutable deployment identifier for deploys, and a bounded search record for “unused.” A useful follow-up is to show the missing-evidence reason in the handback so reviewers learn what proof was absent.
That's a cleaner bar than "something ran." The hook already sends the agent back with a missing-proof note, but the reason is still generic: nothing successful after the last edit, or not enough distinct lookups. Binding each claim type to a specific evidence form would make that note actually useful.
Command plus exit status is close to what I already accept for tests. For deploys, an immutable deployment id is stronger than a 2xx I happened to fetch — a green CI link can still point at the wrong job. For "unused," a bounded search record (what was searched, with which tools, and where it stopped) is clearer than just counting two lookups.
I haven't tightened the evidence types that way yet. I'll start by making the handback name the proof that was absent, so a reviewer can see the gap without making the hook much stricter.
The "never blocks twice in a row" rule is the part worth copying. In the Stop hook's input, that's the stop_hook_active flag: when it's true, Claude is already continuing because a Stop hook sent it back, so the script should let the turn end. Without that check, a hook that keeps failing can keep the session going in circles.
A shape that pairs well with the claim check: a Stop hook that doesn't read the transcript at all and runs the check itself. For a TypeScript project that's tsc --noEmit. If it fails, the script prints the errors to stderr and exits 2, and Claude gets them as the reason to keep working. It adds a few seconds per stop, but the evidence comes from the hook instead of being searched for afterwards.
The two cover different gaps. The claim check handles "deployed" or "doesn't exist", which no type check can confirm. The type check catches the turn where nothing was claimed, but something still broke.
One practical limit: Stop fires on every finished response, including ones that only answered a question, so a slow test suite there runs for nothing. A fast check (types, lint) in the hook, with the full suite left to CI, avoids that.
A Stop hook running the type check, stop_hook_active guard included, is shown step by step here: youtu.be/bWLf6wf7WkU
That's the right reading of the "never blocks twice" rule: if stop_hook_active is already true, the script should let the turn end. Otherwise a hook that keeps failing will just bounce the session.
The complementary shape is a clean split. The claim check still has to look at the transcript for claims a type checker can't see, like "deployed" or "doesn't exist". A Stop hook that never reads the transcript and just runs tsc --noEmit, then exits 2 with the errors, covers the turn where nothing was claimed but something still broke.
I also agree that Stop is the wrong place for a full test suite. It fires after every finished reply, including a plain question. A fast check in the hook and the slow suite in CI is the split I'd use too.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.