I set up AI agents for small businesses, and the most expensive bugs I've seen so far didn't throw a single error.
13 runs, 8 leads, 5 silent zeros
Day one of a lead-finding agent: it ran 13 rounds and came back with 8 qualified leads. Each round hit 9 companies in total, the dedupe against our old list dropped 1, and every one of the 8 came with an evidence link explaining why it was a prospect. No evidence, no lead.
Looked great. Then I read the run log.
Five rounds in a row had returned nothing. No exception, no warning, no retry storm. The sites it was searching had flagged it as a bot and quietly served empty pages, so the agent did exactly what it was told: searched, found nothing, moved on.
If I'd only looked at the final number, "8 leads on day one" would have gone into a report and nobody would have asked about the gap.
Green dashboards lie in the same way
Two more from the same month:
- A status panel for our model endpoints showed all green while the three accounts behind it had been logged out for two days. The panel checked that the process was alive, not that a request could actually be served.
- A session cookie said it was valid for more than a year. The server kicked it in under 3 days. We switched to opening the real settings page every 4 hours instead of trusting the expiry date.
Same pattern each time: the health check measured something adjacent to the work, not the work.
Old rules keep running
Long-running agents get a standing rulebook. We cap ours at 140 lines of index, and anything over that has to be deleted or merged. One retired rule wasn't fully removed, and a fresh session obeyed it and stopped dead on a sign-up page. It took a while to realize the agent wasn't broken. It was following orders nobody meant anymore.
Count cost per finished task, not per call
Price per token tells you very little. SemiAnalysis's GDPx board (206 expert-level tasks, numbers I read on Nov 9) lists DeepSeek Flash at about $0.94 per task with a 5.4% completion rate, and Opus 5.5 at about $51.41 per task with 12%. Divide cost by completion and you get roughly $17 versus $428 per finished task. Which one is "cheap" depends on whether your job can tolerate failing and retrying.
What I check now
- Every result carries its evidence. Empty results are logged as a result, not skipped.
- Several empty runs in a row get a human look, even with zero errors.
- Health checks hit the real thing: a real page, a real request.
- The rulebook has a hard size cap, and retired rules get deleted, not commented out.
- Each agent reports one number a day: how many things it actually finished, and what each one cost.
What's the quietest failure you've caught in an agent of yours?
Top comments (9)
The blocked-but-empty case is nasty because a block page and a genuinely empty search look identical to the agent. Two cheap guards worth trying: log result count (or raw page size) per source per round and alert when a source that normally returns something hits zero a few rounds in a row, and add one known-good canary query each round. If the canary comes back empty, mark the round as blocked instead of no results, so it never gets counted as a clean zero.
The canary query is a good one. In that run it was 5 empty rounds out of 13 before anyone looked, and a known-good query each round would have flagged the block on round one. Worth stealing.
the 5 empty rounds are the scary part bc the final number still looked fine. travel has the same shape, a hotel sitting in an itinerary looks booked until someone checks it actually is, its why searchspot keeps shortlisted and confirmed as separate things
Yes, that is the part that bugged me most. The day still ended with 8 leads, which looked like a normal number, so nobody asked why 5 of the 13 rounds came back empty. Shortlisted vs confirmed is the same split I was missing: the round ran vs the round actually returned something.
An empty result is a valid return for most tools, so the agent accepts it and moves on. Check the shape at the tool boundary: a count that must be above zero, a required field that must be present. The empty answer then fails on the first call.
Agreed. By its own contract the tool did nothing wrong, empty was a legal answer. The check has to sit where the result comes back. I hit a cousin of this too: a scheduled job reported success every single time while the script never actually started, and an alert channel sat silent from one afternoon to the next day.
the silent zero is why i like tools that return a row saying what happened instead of nothing, a skipped person comes back with a status like no email or over the cap and the agent reads the count of those first. one run of that kind with the inputs is here dev.to/kevinhall/cheap-clay-and-ap...
A status row per skipped item beats a bare count, you can see the why at a glance. Empty and blocked looked identical to my agent because neither one left anything behind. Even a single line saying "blocked, page looked like a bot check" would have saved me 4 rounds.
This highlights one of the most important and underestimated challenges in production AI systems: a successful execution does not necessarily mean a successful outcome. The example of five consecutive empty runs is particularly revealing because the agent wasn't technically malfunctioning; it was operating under misleading environmental conditions and reporting those conditions as legitimate results. That's arguably more dangerous than an exception because conventional monitoring has nothing obvious to flag. I especially appreciate the distinction between checking whether a process is alive and verifying whether it can actually complete the task it exists to perform. This is the difference between infrastructure monitoring and outcome-based observability. The point about measuring cost per finished task is equally important. Token efficiency and low execution costs are meaningless if an agent repeatedly fails to deliver usable results. I'd also argue that confidence thresholds, evidence freshness, and explicit classifications for empty, incomplete, and verified results should become standard parts of agent reporting. The retired-rule example exposes another subtle problem: agent memory and persistent instructions need lifecycle management just like production code, because outdated context can become an invisible source of behavioral failure. Ultimately, dependable agents need more than exception handling; they need mechanisms that question suspiciously successful executions, validate real-world outcomes, and make uncertainty visible. The best monitoring system isn't the one that tells us everything is green. It's the one that tells us when green doesn't actually mean working. Excellent practical lessons for anyone moving AI agents beyond experimentation into production.