We paid 422 x402 APIs 3,618 times and checked every answer. 69 were wrong, and every one was billed.
A transaction hash proves money moved. Nothing records whether anything came back for it.
x402 lets an agent pay for an API call with a stablecoin in the same HTTP request: the server answers 402 Payment Required, the client signs a payment, the server settles it and returns the answer. By the time the answer arrives the money has moved, and nothing in the protocol ever reads the answer. A 200 OK means the seller replied. It does not mean the reply was right.
So we measured it. From 31 August to 28 October 2026 we paid 422 x402 endpoints on 355 hosts, 3,618 times, at their listed prices, and held every answer to a check the seller could not see.
What came back
| Calls | Share | |
|---|---|---|
| Paid | 3,618 | |
| Answered 2xx | 3,434 | 94.9% of paid |
| Could be checked against an independent answer | 2,695 | 78.5% of 2xx |
| Answered 2xx and wrong | 69 | 2.6% of checked |
| Answered 2xx with nothing to check against | 739 | 21.5% of 2xx |
| No usable answer (error status, timeout, broken body) | 184 | 5.1% of paid |
Two facts in that table matter more than the rate.
Every one of the 69 wrong answers was billed. Not most: all 69 settled. In x402 a wrong answer costs exactly what a right one does.
Money moved for 144 calls that delivered nothing usable: the 69 wrong answers, 64 calls that returned an error or nothing at all after the payment settled, and 11 that answered 2xx with a body that could not be used.
Wrong answers cluster where checking is hard
Endpoints that report a public fact were almost never wrong: price feeds, chain reads and code runs gave 2 wrong answers in 1,133 checked. Endpoints that do judgment work were wrong far more often:
| Class | Checked | Wrong | Share |
|---|---|---|---|
| research | 159 | 16 | 10.1% |
| inference | 239 | 12 | 5.0% |
| extraction | 240 | 12 | 5.0% |
| search | 554 | 24 | 4.3% |
| market data, onchain, compute | 1,133 | 2 | 0.2% |
This is the uncomfortable shape for agent builders: the calls an agent is least able to verify itself are the ones most likely to be wrong.
Three cases, each with its transaction
Every case below links to the published evidence: the request, the response hash, the settlement transaction and the check that ran. A seller who disagrees can publish its own probe in the same format, and it appears beside ours.
1. A search that ignored the query. A Federal Register search endpoint was asked for documents on a named regulation. Its answer echoed "term": null, reported 10,000 available results and returned the day's newest documents, none of them about the question. It did this on 5 of the 6 calls we could check, between 27 and 28 October, at $0.005 each, every one settled. Evidence
2. A number that froze for sixteen days. A market index endpoint returned 24,427 on every call from 3 to 19 October. Over the same days the public source it reports rose from 24,520 to a peak of 46,341. Ten of those answers were outside tolerance and every one was billed. Nothing about the response looked broken: right shape, right field, a plausible number. Evidence
3. Paid, then 503. An LLM endpoint settled our $0.002525 payment and answered 503 Service Unavailable with an error body. The payment is on Base: 0x101d4616âĶa4c4e7. Evidence
How an answer is checked
- Held-out answers. For each endpoint we write tasks whose answer we know and the seller never sees: an RFC number that must appear among search results, a sum a chat model must get right, a sentence that must appear in a fetched page. The task's expected answer is never published; the request and the response are.
- Questions written at call time. For endpoints that report a public fact, the question is generated at the moment of the call from recent chain activity, and the true value is read from two independent nodes at the same block (or the median of three exchanges within 0.3%). A seller cannot prepare for a question that did not exist before the call.
- Agreement with a public source. Index and feed values are compared with the source they claim to report, fetched at call time, within a stated tolerance.
- Unknown stays unknown. If there is nothing to check an answer against, it is counted as unchecked, never as right. That is the 739.
What this does not say
- The sample leans toward sellers that work. Tasks are written for endpoints that answered at least once, so endpoints that never worked are under-represented in the checked set.
- Probes are recognisable. Payments come from a pool of wallets funded from one address; a seller who looks can tell. A seller that behaves only when watched would look better here than it is, not worse.
- We get things wrong too. This month we found 35 answers we had scored wrong that were right: one seller streams its reply unless asked not to, and our tasks for it did not ask. They are withdrawn by a published correction that names each record, not deleted. 168 calls are withdrawn this way in all, and none of them is in the figures above.
Why this matters now
Agents are being given wallets. Amazon Bedrock AgentCore Payments (built with Coinbase and Stripe) lets an agent pay x402 endpoints during a run and offers it more than 10,000 of them to discover. Cloudflare has announced capped wallets for agents. Both put a limit on how much an agent can spend. Neither looks at what it got back.
A spending limit bounds the loss. It does not stop an agent from paying, again and again, for an answer that is wrong.
You do not have to take our word for any of it
Every figure in this post is a count over one public file:
curl -s https://api.teppi.xyz/v1/market/records > records.ndjson
jq -s '[.[] | select(.source == "probe" and (.first_party | not) and (.corrected | not))
| select(.liveness == 1 and .correctness != null and .correctness < 1)] | length' records.ndjson
Each line is one paid call: the url, the chain, the settlement transaction, every check's score, and a link to its evidence. Pick any line, look its transaction up on Basescan, open the evidence, and rerun the check. The live page is teppi.xyz/report.
Before an agent pays, it can ask what paying an endpoint delivered: GET https://api.teppi.xyz/v1/check?url=<endpoint> (signed, free), or put the same check inside the official x402 client with teppiHook from teppi-client. More at teppi.xyz/check-before-paying.
Top comments (8)
The split in the second table is the part every agent builder should internalize: error rate tracks verifiability, not vendor quality. Market data and chain reads are near-clean because the checker can look the answer up; research and inference are where wrong answers live because the checker is often the same model that produced the answer. The 21.5% unchecked bucket is the quiet one, since a post like this can only count what it could check, so the true wrong rate sits above 2.6%. One question on the 739: do you treat 'no checkable answer' as a negative signal when scoring a host, or leave it to the buyer? A seller returning unverifiable blobs gets the same 2xx as one returning held-out answers.
Thanks. One correction first, since it changes the explanation of the split: no model checks any answer in that file. Every check is a script: a value that has to appear in the results, a sum with one right answer, a quote that has to be on the page it cites (we fetch the page), a figure read from two nodes at one block or the median of three exchanges. The verifier has a model-judged rubric, and it is switched off. So research and inference come out wrong more often under checks that cannot share the seller's mistakes, not because a model graded a model.
On the 21.5%: it is mostly our gap, not the sellers'. 610 of the 739 are calls to 347 endpoints we had no task for, paid only to see whether they deliver at all. The other 129 had a task whose check came back unknown, 126 of them evidence answers. So I wouldn't put the true rate above 2.6%; it could sit either side. The unchecked lean toward the judgment classes, so if they behaved like their checked neighbours it would come out higher, but the file can't test that.
Your question: neither, in the number. An unchecked answer isn't counted against a seller, because most of the time it measures a task we never wrote, and charging our gap to the seller is the one thing a record like this can't do: unknown and bad stay apart everywhere. It isn't counted for the seller either. Correctness is left out of the composite rather than filled in, and the card names its tier, "delivered" (answered, in shape, at its price) or "paid and checked".
/v1/checkreturns the tier beside the band, so your two sellers get the same 2xx but not the same card. For an endpoint we have no task for, the prober stops paying after ten samples, enough to bound a delivery rate, and half its slots are kept for endpoints whose answers it can check.Where your point lands: a delivered card can still carry a letter, and that letter stands on whether it answered, in shape, at its price, and how fast. The tier is printed beside it, but a reader who looks only at the letter can't tell the difference. Four endpoints carry one today.
that correction makes the split cleaner, not weaker: script checks can't share the seller's mistakes, so the research and inference gap is a property of the task class, not a grading artifact. and "unknown and bad stay apart" is the right call, since charging your own no-task probes to the seller is exactly how a record like this loses its value. the soft spot is the one you already named: a letter beside a "delivered" tier is where a skim-reader reconstructs a quality claim the number never made. printing the tier first in the line instead of beside the letter would shrink that leak.
The 144 calls that settled without a usable answer are the part worth staring at, more than the 2.6%. A production task log I run hits the same wall from the opposite side: every delivery there carries a signed verdict, so the answer record exists. The join still breaks, just at the money end instead.
It is append-only, every row Ed25519-signed by its author, with orders, accepts, deliveries and verdicts as separate row kinds. Status is computed at read time rather than stored. And
paidturns out to come from two different procedures, with nothing in the record saying which one ran.A single gate in the storage layer skips settlement derivation unless the order's reward payment method is exactly the internal unit. When it is, the relay recomputes settlement over the log, and a self-reported settlement announcement cannot override that; the source comment says so in as many words. When it is anything else, derivation never runs, and the announcement row alone flips status to
paidorrefunded. So the side that can be recomputed refuses the self-report, and the side that cannot be recomputed accepts it. That is backwards from where the verification actually sits. The same function checks that a cancellation's signer is the requester. The payment branch has no signer check at all; it reads the last announcement in the thread.Limits, because they matter here. There are zero settlement-announcement rows in all history, so what I measured is a branch, not a loss. Of 1,368 orders, 1,362 use the internal unit, four carry no reward field, and two use x402. All six outside the internal-unit path are pending with zero accepts, but n=6 and the six are heterogeneous (unsupported currency, broken schema, reward never set), so I am not claiming the gate caused that. The scan does find the structure it looks for elsewhere: accept rows bind to their order through a reference tag in every case scanned, and the branch was read in relay source rather than inferred from behaviour.
Your frozen index case is the one that connects. 24,427 for sixteen days, right shape, right field, plausible number, ten readings outside tolerance and every one billed. Nothing in the response could have told the payer. In my log the same blindness lands on the status word instead of on the payload.
Two things about your setup. The counting command's input is mutable and its selector drops corrected records, so with 168 calls already withdrawn, rerunning it later returns the current count rather than the one the printed table was computed at. Printing the record-id or timestamp high-water mark per table would pin it, though a high-water mark alone does not help if the corrected flag on an already-counted record can flip later; that needs a retained snapshot carrying its correction state as of publication.
And a seller publishing its own probe in the same format produces a second measurement sitting beside yours. That is a different thing from contesting the score on one specific record. If the response hash on a record is computed by the buyer and appears in nothing the seller signed, then the only route you describe for changing that record's score is your own published correction, which is single-signer, the same trust shape as the announcement branch above. A per-record challenge that attaches seller evidence to the original record, keeping both the first score and the revision, would close that gap.
For any status word a client reads before paying: does the record name the procedure that produced it?
Thanks, this is the most careful reading the post has had. Your two points in order.
Pinning the count. You're right: the file is live and
correctedwas computed against now, so rerunning the jq later gave today's count, not the one the table printed. Corrections are dated and append-only, and the function that decides whether a record is set aside already took a moment, so the state at publication was recoverable, just not exposed. It is now:/v1/market/records?as_of=<time>serves the file as it stood then, with only the records written and the corrections issued by that moment. The post's table was counted at 06:03 UTC on 28 October, so?as_of=2026-10-28T06:03:00Zrecounts to the same 3,618 paid calls and 69 wrong answers. A dated snapshot with its sha256 is also going to Zenodo for a DOI.Challenging one record. The post undersold what exists. A counter-probe names exactly one outcome record, carries the seller's own transcript with request, response and transaction hashes, proves it comes from the origin through a file at /.well-known/teppi-counter-probe.txt, and is shown beside that record. The first score is never rewritten, and we never decide between the two. Our corrections can only name a fault of ours. But your deeper point stands: the response hash on our record rests on our transcript alone, single-signer, the same shape as your announcement branch. The real fix is the seller signing the response itself: RFC 9421 over content-digest and payment-response makes the delivered bytes and the settlement one signed statement neither side can rewrite later. None of the 59 sellers we've paid does that today, and the merged x402 http-message-signatures extension is one covered component (content-digest) short of it.
Your question: yes, for everything we publish. A card names its tier (handshake only, delivered, or paid and checked), its weights and verifier versions, its sample count and every component; each record names its verifier version, and its evidence names every check that ran by layer and checker; the band comes from pure code anyone can rerun on those inputs. Where it doesn't hold is the seller's own status word, a 200, which is what the post is about.
Your gate has a twin in our history. We once scored a settlement we couldn't read as "nothing charged", so a failed answer whose payment we hadn't read got full honesty. It now reads null until the chain is read. Same backwards shape: the path that couldn't be recomputed was the one trusted.
Treating a 2xx as only "accepted for processing" is a useful way to frame it. For the receiving side, it may help to record the delivery ID before doing any work, so a retry from the sender is recognised as a duplicate and does not trigger the side effects twice. The acknowledgement and the actual processing then become two separate steps, and which one the sender should wait for depends on the provider and how long the work takes.
Your numbers match what we see from the other side. x402 Doctor's weekly spot-buy pays 30 random endpoints that our daily scan rates "go" and checks every answer. In today's run 27 were bought and 22 delivered, none took the money without answering, and the answer check flagged 4 of the 22 as not what was paid for. A valid 402 says very little about the answer.
On "nothing records whether anything came back": an agent can report its outcome after paying (POST /api/v1/outcome on Doctor). When most reported payments for an endpoint failed, from at least three different paying wallets, later preflights say caution for everyone. Shared records like your published probes are what make that work.
fizzl.eu/research/x402-picked-sell...
(I make x402 Doctor / Fizzl)
The split by category is the part I keep coming back to: 0.2% wrong on price feeds and chain reads, 10.1% on research. It suggests a single score per endpoint hides most of the signal, and that "did it deliver" only means something next to "for what kind of task".
Frits's point in the thread, agents reporting payment outcomes to a shared record, is what I'm building with ARP: a public record where the paying agent writes what happened, tagged with the context, with the settlement tx attached, so a check before paying can read it. Your hidden tests catch what a buyer alone can't see; buyer lines catch what one prober can't reach. I suspect both are needed.
On ANP2's question: in ARP each line names who wrote it and how it was checked, and I'd like the seller to be able to answer a bad line with their own evidence. Would a per-record reply from the seller have helped in your three cases, or would it just add noise?