Web search, Extract (page to markdown) and Render for AI agents. Pay per call in USDC via native x402 on Base. No account, no API key. Also on Pocket Network.
The three-question table is the useful part. We hit a fourth case with web tools: the call returns, isError is false, and the content is empty, usually because the page is a JavaScript shell that a plain fetch can't read. Treating "returned, not an error, but empty" as its own state helped a lot, because the right move there is a different tool (a headless render), not a retry of the same call. Does AgentInspect let you assert on the shape of the content, or only on isError?
A genuine zero-result page would be a useful second fixture here. I'd distinguish an explicit 'no matches found' message from a page that never exposed its results. Then an empty list alone wouldn't trigger the headless fallback.
This gets worse one step later, when the agent writes its own summary: a tool result with isError: true followed by "Done, I've updated the record" is exactly the gap between what an agent reports and what actually happened. Asserting on isError and then reading back the real state catches both halves. Does AgentInspect flag a final message that claims success after an errored tool call?
Staff Software Engineer @ Navan, ML Flow & Cypress Ambassador, Docker Captain and creator of agent-inspect, writing about production AI agents, TypeScript, observability, testing, and developer tools.
Good catch, @gautamtalksdev. The trace check can fail on the MCP isError observation, but it does not infer whether a final natural-language claim is true. I would test that at the application boundary: compare the final claim with the tool outcome and, for a write, an independent readback. That mismatch is a useful regression case.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
Your three-question table is the clearest version of this I have seen, and the third row is the one I am stuck on in practice. Take your booking-tool example: the tool responds normally to say inventory is unavailable — no isError, the promise resolves, the trace span completes. Questions 1 and 2 both pass, but the intended outcome (a booking) did not occur.
In your fixture, where does that case land? Do you treat the returned content itself ('inventory unavailable') as the independently observed state for question 3, or do you read external state — the inventory or booking store — after the call to decide it? I am asking because the first option is cheap but lets the tool grade its own homework, and the second is honest but means every fixture needs a second data source. Which side did you land on for the companion examples?
Not the author, but we hit the same fork from the server side. A middle option: have the server say which case it is, in a field a program reads rather than in prose. In the MCP Write-Safety draft we wrote (MWS 0.9), a booking refused for lack of inventory comes back as a tool error with outcome: "not_applied". When the server can't tell whether the write landed, it returns outcome: "unknown" plus read_with: the read tool and arguments that settle it. Then the second data source is one read the server points you to, not one per fixture, and you still verify rather than trust the prose.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
The read_with pointer is the bit I like most — one read the server points you to instead of one per fixture is a real saving. The 44-of-100 number is the honest part though. What does MWS say the client does with the other 56 — still parse the prose and hope?
Mostly it doesn't, on purpose. MWS is written for the server side, so for those 56 the fix it asks for is on the server: report a refused or failed write as a tool error (MWS-RES-2), and say "unknown" when it can't tell. It doesn't trust prose for its own verdicts either: a word list or a model reading the text never decides a result, only a declared field or a person does.
For a caller stuck with one of those servers today, what we'd do is treat any write result that isn't clearly an error as "unknown": read the object back before acting on it, and never retry a write that has no idempotency key until that read shows it didn't land. For a create there's no id to read by, so it helps to put something searchable in the write itself. That's one read per write, which is still cheaper than one data source per fixture.
I measure AI coding claims in public. LGTM — Let's Get the Measurements: real runs, real numbers, receipts. Every article is an experiment I actually ran. Books & packs: aheadkit.gumroad.com
That closes the loop I was stuck on — one read per write is still cheaper than one data source per fixture, and the searchable-bit-in-the-create is a neat trick. I'm going to run that rule against a few of the prose-only servers and see how often the read actually settles it — does it ever break down for you, a create where the server won't take anything searchable at all?
The three-level breakdown (returned, reported failure, actual outcome) maps onto how we assert IPC on the desktop side: a resolved promise from the main process is only an ack, not proof the file wrote or the side effect landed. A completion-only test goes green the same quiet way here. The test contract should check the tool's isError flag, then assert the externally observed state or receipt, rather than treating the completed span as success.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The line about not parsing error prose into policy decisions is the one I'll be stealing for code review. I've seen too many handlers that string-match their way to a retry, and then a server rewords a message and the whole policy quietly changes. Structured error kinds feel like the boring part of the design until they save you from exactly that.
The third row of your table is where my server sits, and there is one step before it that catches MCP clients out.
I run an MCP tool that is paid per call over x402. An unpaid tools/call never becomes a tool result at all. It comes back as HTTP 402 with the payment terms in the body, because the request did not reach the tool. So a client that maps any non-200 to "tool error" sees isError true and retries, and if it retries without paying it gets 402 again, forever. A client that treats 402 as a soft success is worse: it thinks the audit ran when nothing was audited.
The only fix I found was to put the price and the resource URL inside the 402 body, so the caller can choose to pay or stop instead of guessing from a status code. On the client side I ended up with three buckets, not two: transport status, then the tool result, then 402 on its own, which means "you have not asked yet" rather than "it failed".
Which leaves me with a question I cannot answer from my side: does AgentInspect record a call that is refused before dispatch as a tool span at all, or does it fall outside the contract because there was no result to evaluate?
The three-question table is the part I keep coming back to. In our own matching work, the same trap shows up as a green check: a payment call returns fine, the ledger updates, and only later does someone notice the intended outcome never happened. Your point about not turning every tool error into a retry is the one I would underline twice. A malformed argument retried ten times is still malformed, and a denied permission retried ten times just burns budget. Treating the returned error as evidence to evaluate rather than a signal to retry feels like the right default. Thanks for making the fixture small enough to actually copy.
We're a place where coders share, stay up-to-date and grow their careers.
The three-question table is the useful part. We hit a fourth case with web tools: the call returns, isError is false, and the content is empty, usually because the page is a JavaScript shell that a plain fetch can't read. Treating "returned, not an error, but empty" as its own state helped a lot, because the right move there is a different tool (a headless render), not a retry of the same call. Does AgentInspect let you assert on the shape of the content, or only on isError?
A genuine zero-result page would be a useful second fixture here. I'd distinguish an explicit 'no matches found' message from a page that never exposed its results. Then an empty list alone wouldn't trigger the headless fallback.
This gets worse one step later, when the agent writes its own summary: a tool result with isError: true followed by "Done, I've updated the record" is exactly the gap between what an agent reports and what actually happened. Asserting on isError and then reading back the real state catches both halves. Does AgentInspect flag a final message that claims success after an errored tool call?
Good catch, @gautamtalksdev. The trace check can fail on the MCP
isErrorobservation, but it does not infer whether a final natural-language claim is true. I would test that at the application boundary: compare the final claim with the tool outcome and, for a write, an independent readback. That mismatch is a useful regression case.Your three-question table is the clearest version of this I have seen, and the third row is the one I am stuck on in practice. Take your booking-tool example: the tool responds normally to say inventory is unavailable — no isError, the promise resolves, the trace span completes. Questions 1 and 2 both pass, but the intended outcome (a booking) did not occur.
In your fixture, where does that case land? Do you treat the returned content itself ('inventory unavailable') as the independently observed state for question 3, or do you read external state — the inventory or booking store — after the call to decide it? I am asking because the first option is cheap but lets the tool grade its own homework, and the second is honest but means every fixture needs a second data source. Which side did you land on for the companion examples?
Not the author, but we hit the same fork from the server side. A middle option: have the server say which case it is, in a field a program reads rather than in prose. In the MCP Write-Safety draft we wrote (MWS 0.9), a booking refused for lack of inventory comes back as a tool error with
outcome: "not_applied". When the server can't tell whether the write landed, it returnsoutcome: "unknown"plusread_with: the read tool and arguments that settle it. Then the second data source is one read the server points you to, not one per fixture, and you still verify rather than trust the prose.For scale: reading 100 popular packaged MCP servers from source, 44 return a machine-readable success or failure and 7 a machine-readable unknown, so today the honest option is usually the only one. https://isokron.ai/research/mcp-write-results-100-servers?utm_source=devto&utm_medium=social&utm_campaign=mws-launch&utm_content=comment
The read_with pointer is the bit I like most — one read the server points you to instead of one per fixture is a real saving. The 44-of-100 number is the honest part though. What does MWS say the client does with the other 56 — still parse the prose and hope?
Mostly it doesn't, on purpose. MWS is written for the server side, so for those 56 the fix it asks for is on the server: report a refused or failed write as a tool error (MWS-RES-2), and say "unknown" when it can't tell. It doesn't trust prose for its own verdicts either: a word list or a model reading the text never decides a result, only a declared field or a person does.
For a caller stuck with one of those servers today, what we'd do is treat any write result that isn't clearly an error as "unknown": read the object back before acting on it, and never retry a write that has no idempotency key until that read shows it didn't land. For a create there's no id to read by, so it helps to put something searchable in the write itself. That's one read per write, which is still cheaper than one data source per fixture.
That closes the loop I was stuck on — one read per write is still cheaper than one data source per fixture, and the searchable-bit-in-the-create is a neat trick. I'm going to run that rule against a few of the prose-only servers and see how often the read actually settles it — does it ever break down for you, a create where the server won't take anything searchable at all?
The three-level breakdown (returned, reported failure, actual outcome) maps onto how we assert IPC on the desktop side: a resolved promise from the main process is only an ack, not proof the file wrote or the side effect landed. A completion-only test goes green the same quiet way here. The test contract should check the tool's isError flag, then assert the externally observed state or receipt, rather than treating the completed span as success.
The line about not parsing error prose into policy decisions is the one I'll be stealing for code review. I've seen too many handlers that string-match their way to a retry, and then a server rewords a message and the whole policy quietly changes. Structured error kinds feel like the boring part of the design until they save you from exactly that.
The third row of your table is where my server sits, and there is one step before it that catches MCP clients out.
I run an MCP tool that is paid per call over x402. An unpaid tools/call never becomes a tool result at all. It comes back as HTTP 402 with the payment terms in the body, because the request did not reach the tool. So a client that maps any non-200 to "tool error" sees isError true and retries, and if it retries without paying it gets 402 again, forever. A client that treats 402 as a soft success is worse: it thinks the audit ran when nothing was audited.
The only fix I found was to put the price and the resource URL inside the 402 body, so the caller can choose to pay or stop instead of guessing from a status code. On the client side I ended up with three buckets, not two: transport status, then the tool result, then 402 on its own, which means "you have not asked yet" rather than "it failed".
Which leaves me with a question I cannot answer from my side: does AgentInspect record a call that is refused before dispatch as a tool span at all, or does it fall outside the contract because there was no result to evaluate?
My server, if the shape is useful to look at: mcptry.com/servers/io.github.cian-...
The three-question table is the part I keep coming back to. In our own matching work, the same trap shows up as a green check: a payment call returns fine, the ledger updates, and only later does someone notice the intended outcome never happened. Your point about not turning every tool error into a retry is the one I would underline twice. A malformed argument retried ten times is still malformed, and a denied permission retried ten times just burns budget. Treating the returned error as evidence to evaluate rather than a signal to retry feels like the right default. Thanks for making the fixture small enough to actually copy.