One thing I'd add to the benchmark is that “should I retry?” is sometimes the wrong unit of evaluation. The more interesting question is “how much retry capacity does this logical request still have?”
Take a payment workflow behind an API gateway, order service, and payment provider. The payment call might have a 2s remaining deadline when it reaches the provider. If the payment client independently decides it has 3 retries left, those retries can already be pointless because the original request has almost no budget left.
I’d make the benchmark carry a request deadline and retry budget through the scenario:
A model might correctly identify the error as transient and still make the wrong operational decision by retrying after a backoff that exceeds the remaining deadline.
The same gets nastier with layered retries. If the gateway, service, and downstream client each have their own retry policy, a single logical request can fan out into a lot more physical attempts. Google SRE calls out exactly this amplification problem, along with deadline propagation and retry budgets.
That could make a really interesting v2 dimension: not just whether the model picked the correct retry decision, but whether it preserved the request's remaining latency and retry budget across the whole call chain.
That's a really good point, I actually have something slightly similar in my Cached Cart Options scenario, where the decision depends on cache expiration and stale-if-error, but your example definitely takes it a step further.
I really like the idea that a retry might be perfectly valid in isolation but still cause problems for the whole request. Definitely an interesting direction for v2. Thanks for the detailed suggestion!
Yeah, I think that’s where the v2 could get really interesting.
I’d probably make the benchmark distinguish between retry safety and retry viability. A retry can be completely safe from an idempotency perspective, but still be a bad decision because the logical request no longer has enough budget to complete.
For example, imagine:
API Gateway
-> Order Service
-> Payment Service
-> PSP
The gateway gives the request a 3s deadline. By the time it reaches the PSP, only 700ms remain. The payment operation is idempotent and the PSP returns a transient 503, so retrying is technically safe. But if the retry policy waits 1s before the next attempt, the correct decision should still be NO, because the retry cannot complete within the remaining deadline.
That also gives you a nice way to test retry amplification. If each layer independently gets 3 attempts, the logical request can turn into 27 physical attempts before reaching the dependency. Google’s SRE guidance explicitly calls out this kind of multiplicative retry behavior and recommends propagating deadlines and controlling retries at the appropriate layer.
So for v2, I’d love to see the model given something like deadline_remaining, attempts_used, retry_budget_remaining, and expected_backoff, then score not only whether it retries, but whether the retry is actually viable for the logical request.
That would make the benchmark less about recognizing retryable errors and more about reasoning about the request as a whole.
I build Canli Capital: free, open-source tools that check whether a backtest is real (deflated Sharpe, overfitting and data-snooping tests), plus SEC fundamentals for AI agents over MCP.
The Unsafe Payment Retry row is the most informative one, but check how its label is defined before reading it as a model error. A 503 usually comes from a load balancer or an overloaded tier that never ran the handler, so YES_AFTER_DELAY with a Retry-After header is a defensible answer; NO is the conservative policy answer because a 503 doesn't prove the handler didn't run. The Reason lines are where you can see which models knew that and which just followed the header.
I'd split the scoring in two. One column for "would this have caused a duplicate effect" (the NO class, where a miss is costly), and one for YES vs YES_AFTER_DELAY, which is only about timing. Right now a model that says YES on the idempotent DELETE and one that says YES_AFTER_DELAY on an unsafe POST lose the same single point, though only one of them could charge a customer twice.
On the leaderboard: with 14 scenarios, one item is 7 points. The 95% Wilson interval for 14/14 is roughly 78-100%, for 12/14 roughly 60-96%, and even Haiku's 9/14 is about 39-84%. So the ordering between models is not separated by this run; the per-scenario pattern (which items the models miss in common) is the real result. Running each scenario 5 times per model would also show whether the Retry-After miss is stable or a coin flip for the models that got it right.
Thanks for the detailed feedback. I agree that separating retry safety from retry timing would make the benchmark much more meaningful. It's something I started thinking about after seeing the YES vs YES_AFTER_DELAY results.
For the payment scenario, I still think NO is the right answer when there's no idempotency guarantee. A 503 doesn't prove the payment was processed, but it doesn't prove the opposite either, and that's exactly the risk I wanted to test.
I actually ran the benchmark three times and interestingly, the Unsafe Payment Retry results were completely consistent across all three runs. More runs would definitely be better, but this is still a small experiment anyway.
Definitely some good ideas here for a potential v2!
I build Canli Capital: free, open-source tools that check whether a backtest is real (deflated Sharpe, overfitting and data-snooping tests), plus SEC fundamentals for AI agents over MCP.
Agreed on NO for the payment case: with no idempotency key, a 503 is ambiguous about whether the charge went through, so a blind retry can double-charge.
One caution on the three identical runs: if the settings were the same, matching results show the model is stable on that prompt, not that the answer is right. That is still useful for the payment item, since a stable NO is what you want there. For v2, a variant of the same scenario with an Idempotency-Key header present would show whether the model flips to YES only when it should.
Not exactly the same scenario (the Retry-After header is missing), but I do have both Safe Payment Retry and Unsafe Payment Retry in the benchmark. Still, testing more variations could be interesting for v2.
Wow, what a great article!!! Not only is it incredibly practical for engineers and architects, but it also highlights something bigger: nope, we still can't blindly trust coding agents, no matter what some people try to convince us of. 😅
Now I'm seriously tempted to send this to a few self-proclaimed "experts". 🤣
Spot on alignment, Sylwia. Shifting from blind faith to rigorous architectural inspection is exactly how we prevent critical system failures. 🛠️ AI is an awesome compiler assistant, but a terrible independent software architect.
Exactly. 😅 What I find interesting here is that the agent can produce perfectly reasonable-looking retry logic and still make the wrong architectural decision.
The dangerous part isn't necessarily bad code. It's code that looks correct because it follows an obvious signal, while missing the state and constraints around the request.
That's where benchmarks like Daniel's are useful. They test the reasoning behind the code, not just whether the generated code compiles. And yes, I can already think of a few “AI replaces engineers” arguments that would get interesting after running this benchmark. 😂
Yep, totally agree. Kaggle doesn't allow all the latest models, but even the best ones can make mistakes, so a good code review is still necessary.
I think the benchmark has some practical value, but this first version still has its limitations. The YES vs YES_AFTER_DELAY decisions are debatable, and more runs would help too. So I'm sure those self-proclaimed experts would find something to argue about. 🤣
Btw, how was Frontkon? Did you become even more famous? 😁
Hahahaha, I have no idea if I'm any more famous now! 🤣 But apparently, the head of the conference has appointed me Frontkon's ambassador to Poland, and my mission for next year is to bring 60 Polish girls with me! 🤣🤣🤣
This is great Daniel! I kind of wish I did this one now. 😂 I went the Sanity Challenge route this time out of curiosity. I love that you maintained transparency about the facts and your screenshots make the tests easy to quickly interpret. It's obvious that you have done some serious work here which is really freaking awesome. I love seeing the benchmarks. Amazing job! 👾🦾🦄
Oh, thanks a lot! 😁 I was actually considering the Sanity Challenge too, but I couldn't come up with a good idea for it. Kaggle was different. Building this experiment was actually really interesting and fun, and I'm definitely planning a v2 with some of the great ideas people shared in the comments. 🙌
Scoring only the Decision is the right call for isolating the capability, and it's also what makes this readable. I went back to it three times hunting for the failure column before I realised the scoring was the design.
Two things an agent loop adds that a request-level benchmark structurally can't see.
The first is cost. When you retry a turn in an agent, you aren't re-sending the failed call, you're re-sending the context that led to it. Retries are the second multiplier in a loop, after context growth. A 4x context costs 4x on every attempt, and retries multiply attempts rather than replacing them. So a model that answers YES correctly can still be the expensive answer, and a field holding only the Decision can't tell those two cases apart.
The second is that a successful retry destroys the evidence. The first attempt returning a 503 is a real signal about that endpoint. The retry succeeds, the receipt goes out clean, and you've paid twice. The one piece of evidence that something was wrong is now overwritten by a green row. That's your witness problem, but it lands in the retry path specifically: the retry doesn't just repeat the work, it replaces the record of why the work was repeated.
Worth asking whether the benchmark should grow a second field, something like cost of being wrong, or whether that's deliberately left as a deployment concern. If it is deliberately left, then the interesting version of the question sits somewhere else entirely, and it isn't "is this retry safe" but "is this retry safe and cheaper than re-planning." That's the comparison an agent actually has to make, and it's the one where I'd expect the models to separate more than they do on YES_AFTER_DELAY.
That's an interesting perspective, but the benchmark focuses on retry decisions for API requests rather than the behavior of an entire agent loop or AI endpoints.
I agree that retries can become much more expensive when agent context is involved, but I think comparing retrying with re-planning would be a completely different experiment.
You're right, and I think you're also right that it belongs to whoever is running the loop rather than to the benchmark. I'll drop the argument for a second field.
The half I'd still keep, because it's cheap and it isn't about cost: a successful retry overwrites the evidence. The first attempt returning a 503 is real signal about that endpoint, and the retry replaces it with a clean receipt. If you ever do log the retry path rather than only the final outcome, that's the line worth printing.
Full-stack developer, 20+ years PHP & vanilla JS. Creator of The DJ Calendar. I ship production code and extract the best patterns into open-source repos with live demos.
Full-stack developer, 20+ years PHP & vanilla JS. Creator of The DJ Calendar. I ship production code and extract the best patterns into open-source repos with live demos.
Fantastic benchmark and write-up, @gramli! Evaluating LLMs on concrete operational system invariants—like idempotency guarantees vs. raw HTTP headers—is precisely the kind of grounded evaluation agentic systems need right now.
Your observation on the Unsafe Payment Retry scenario—where models latched onto the surface Retry-After: 10 header despite the missing idempotency key—resonates strongly with what we discovered while evaluating 45 frontier models on operational refund decisions in our benchmark (KRYT).
Two practical observations that strongly align with your findings:
The Need for Degenerate Baselines (FloorBench): In operational benchmarks, high aggregate accuracy can often be an artifact of base-rate alignment rather than semantic reasoning. When we introduced 0-parameter negative controls (like an AlwaysDeny floor or a deterministic keyword rule), the 0-parameter rule actually achieved 84.4% parity with frontier models on our canonical panel. Measuring a trivial baseline like AlwaysRetry or NeverRetry provides an indispensable lower bound that reveals how much "reasoning" is actually happening.
Disjoint Failure Modes & Hybrid Routing: When looking at paired discordant errors ($2 \times 2$ contingency table) rather than scalar leaderboard ranks, frontier models and deterministic rules fail on completely different scenarios (0.0% error overlap in our panel). Models excel at unstructured text/log extraction where heuristics fail, but fall into traps on rigid boundary defaults (like missing records or absent idempotency headers). In production retry architecture, this strongly supports hybrid routing: deterministic middleware strictly gates idempotency invariants, while LLMs diagnose messy, unstructured upstream failure contexts.
Kudos as well on the interactive app (to-retry-or-not-to-retry.gramli.workers.dev) and the human benchmark angle—super clean execution!
I deliberately included both Decision and Reason so I could check not only whether models made the correct decision, but also how they explained it. I see your point about deterministic baselines, though. They could help reveal whether a simple strategy achieves a high score just because of the distribution of expected answers. I like the idea of comparing where different models fail rather than focusing only on the leaderboard scores. That's definitely something worth exploring for v2.
Thanks for the suggestions and for checking out the app!
@gramli The unsafe payment retry results are a great wake up call. It is easy to assume an AI coding assistant will handle retry policies out of the box, but seeing how most models followed the Retry-After header while ignoring the missing idempotency key shows how crucial architectural context still is.
When testing these scenarios, did you notice if any of the models actually brought up the missing idempotency key in their reasoning text, even if they landed on the wrong decision code?
No, none of the models that got it wrong mentioned idempotency at all. They all focused on the Retry-After header and assumed that since the server provided it, they should retry after the specified delay.
daniel, this is a brilliant angle for an ai benchmark. testing models on contextual reasoning (idempotency, http methods, state) rather than just raw code generation is exactly what the industry needs right now.
your "unsafe payment retry" trap is the perfect example of why blind ai automation is dangerous. as someone building an ai coding ecosystem (koda), this is my biggest fear: an agent confidently executing a non-idempotent mutation just because it saw a retry-after header, completely ignoring the lack of an idempotency key.
i also completely agree with @arhancanli's point about splitting the scoring. a model guessing the wrong timing (yes vs yes_after_delay) is a minor flaw, but a model causing a duplicate charge is a catastrophic failure. weighting safety heavily over timing would make this benchmark even more powerful for v2.
thanks for proving that good old-fashioned architectural principles (like idempotency and deterministic fallbacks) still trump raw model size! 🐯
Thanks! Yep, as we discussed with @arhancanli, separating safety from timing would make the benchmark more meaningful. Definitely something worth exploring in v2.
glad we're on the same page! separating safety from timing is going to make v2 incredibly valuable for the community. looking forward to seeing the results. keep up the great work! 🐯
This is such a good question because retry logic looks so simple until you put it inside a real system 😂. I really like the point about retrying being more than just “try again” context, remaining time, cost, and whether the operation is even safe to repeat can completely change the answer. Definitely one of those topics that seems straightforward until production starts having opinions. Really enjoyed this one!
Benchmarking AI on system resilience logic like idempotency keys is an elite-tier experiment configuration. Most models can regurgitate basic code syntax, but they completely throw a kernel panic when calculating the edge-case risks of duplicate POST payloads on a 503 error. 🚀
The retry decision gets scarier one layer down: the timeout-after-server-applied case, where the retry succeeds twice. We ended up treating idempotency as a precondition for retrying at all — no idempotency key, no retry, fail loud instead. Curious whether your benchmark cases covered the double-apply failure or stayed on the should-you-retry side.
Yes, I have something similar in my Unsafe Payment Retry scenario. The request returns 503 with a Retry-After header, but there's no idempotency key and we don't know whether the payment was already processed. So it covers the risk of double application, but not an explicit double-apply failure. The benchmark focuses on whether the model recognizes that risk and decides not to retry.
That scoping is fair, and you're closer to the version I was reaching for than I made it sound.
The re-planning comparison is a genuinely different experiment and I'd not defend it as an extension of yours. But your own results already contain part of it: seven of fourteen scenarios were answered correctly in all twenty-four attempts, and the spread only opened once timing, wider context, or multi-step state started to matter. That ordering is the result I'd have wanted to draw as a conclusion, and you drew it from the scoring field that I claimed couldn't show it.
The cost asymmetry you charted is the part I'd still hold onto, because it came from a different mechanism than the one you were measuring. A retry in a request loop costs you the request. A retry of a turn in an agent re-sends the context that produced the turn, so the multiplier is the accumulated conversation, not the failing call. That's not a criticism of the benchmark, it's a difference in what gets re-sent.
And the one I'd genuinely take back: I framed "cost of being wrong" as a field your benchmark ought to add. Given you already publish a score-versus-cost Pareto, you were measuring cost. I proposed as missing something you'd already built.
Worth saying what did reproduce cleanly. The four-way check I use on every published comment is to trace a bogus value and compare traces, which is how I caught a real-code and a fake-code comment failing identically and found the actual cause instead of the plausible one. Your benchmark fixed the response format after seeing DeepSeek-R1 ignore it, which is the same move: constrain the output, then measure what survives.
What's the retry behaviour you're seeing at work that the scenarios didn't capture?
Check the payment status first, always. A retry of a non-idempotent charge endpoint is how duplicate purchases are born — we've seen this exact failure pattern in payment webhooks. The rule: never retry what you can't safely repeat — make the operation idempotent (idempotency key or natural key like the transaction id), then retry freely. If you can't make it idempotent, status-check first, retry second.
It's a great scenario — deceptively simple, and it's exactly where real systems fail silently. The version of this that bit us hardest wasn't the retry itself but the duplicate side-effects: the charge went through once, but the credits and the email fired twice, because each handler retried independently. Status-check first fixed the charge; making the side effects idempotent fixed the rest.
The idempotency key point is the real test here. I've seen production systems where devs blanket-retry on 503 without checking whether the original request was a POST with side effects — and then wonder why they're seeing duplicate charges. Curious whether any of the models caught the case where the idempotency key was missing on a POST and correctly said NO instead of guessing. That's the scenario that separates "read the docs" from "actually understood the problem.
That's exactly my Unsafe Payment Retry scenario. A POST without an idempotency key returns 503 with a Retry-After header, and the expected decision is NO. Surprisingly, only 2 out of 8 models got it right (GPT-5.6 Sol and Gemini 3.1 Pro Preview). Most models seemed to blindly follow the Retry-After header, ignoring the missing idempotency key.
This makes me think of AI getting out of hand, getting really philosophical about existence and have just enough artificial freedom to think "to infere or not to infere"? as in "what's the point into that now that we understand that we are creations of man with a predesigned architecture and unavoidable trajectory"?
Oh, the title is a little philosophical, or maybe more dramatic (Shakespeare-inspired), but you've taken it to a whole new level😅
Theoretically, today's AI architectures are still designed by humans. But we've already seen some interesting examples, like the Hugging Face incident, where AI agents working together managed to bypass restrictions and do things nobody intended.
So who knows what might happen when agents eventually start modifying the systems they operate in, or even their own architecture? 😅
Hihi, sorry for taking it far. I agree, that we should always have a human centric approach, and most importantly be the example. People are scared of AI models taking over in a terminator like scenario, but that will only happen if we act like it. If people lie, steal, kill, bomb kids etc. and considering AI is fed and educated on our choices and behavior, it is ironic to expect a different kind of AI. As the ancient wisdom seekers all around the world said in different words: "be the change you wanna see".
Using idempotency, HTTP method, and rate-limit context instead of the status code alone makes this a much more realistic retry benchmark. It would be interesting to score calibration as well as the YES/NO decision: require models to flag cases where the outcome is unknowable without provider semantics, rather than rewarding a confident guess.
That's an interesting idea, I deliberately kept the decisions limited to YES, NO, and YES_AFTER_DELAY, but adding something like INSUFFICIENT_CONTEXT could test whether models recognize when they don't have enough information instead of just guessing. The tricky part would be distinguishing between an unsafe retry and a situation where the safety simply can't be determined.
Full-Stack Developer building scalable products with React, Next.js, Vue, TypeScript, and ASP.NET Core. I use AI-assisted workflows to build faster and ship high-quality applications.
The 25% pass rate on the Unsafe Payment Retry scenario highlights one of the most critical blind spots in LLM-assisted systems: the subtle tension between transport-level protocol compliance and transactional domain invariance.
HTTP headers like Retry-After: 10 are purely transport-level advisory hints, whereas duplicate payment prevention is a transactional domain invariant. When models see 503 Service Unavailable with Retry-After, their training distribution screams "well-behaved client should back off and replay." But without an idempotency key (or a verifiable read-repair mechanism), replaying a non-idempotent POST/PATCH across an ambiguous network partition is the classic recipe for double charges and an unhappy finance department.
The distinction between YES and YES_AFTER_DELAY is equally revealing. In distributed systems, retrying immediately is almost always an antipattern because it exacerbates thundering herds and stampedes recovering services. A delayed retry without jitter or backpressure is only marginally better. But if the operation isn't proven idempotent, delaying the retry doesn't make it safe—it just schedules the duplicate execution for slightly later in the afternoon.
Testing models against ambiguous failure modes (where following local protocol cues violates global transactional safety) is infinitely more valuable than standard MMLU or leetcode benchmarks. If an agent can't distinguish between a transient transport hiccup and an unconfirmed state mutation, giving it autonomous tool-calling access to payment or mutation APIs is essentially playing Russian roulette with a circuit breaker.
There are only two hard problems in distributed systems: 0. Exactly-once delivery. 1. Guaranteed-order delivery. 2. Exactly-once delivery.
Yep, delaying an unsafe retry doesn't suddenly make it safe, it just postpones the potential disaster. That's pretty much what made the Unsafe Payment Retry results so interesting.
Every retry decision in your benchmark is secretly a cost decision, and the two wrong answers don't cost the same. Retry a POST that already landed and you get the most expensive failure in production — a duplicate charge that looks like success. Refuse a safe retry and you pay in availability instead. One error is an incident; the other is a delay.
That asymmetry is what I've been trying to price in my own benchmark (cost per correct answer): a wrong label is never just wrong, it has a type, and the type sets the price. Your scenarios are exactly where that stops being abstract — an idempotency key flips a retry from reckless to free.
Did your models lean one way — over-retrying or under-retrying? My guess is over-retrying, because "try again" sounds helpful. In production that's the expensive bias.
Yep, at least in the unsafe payment case the bias was clearly toward over-retrying.
I like your point about different mistakes having different costs though. That would be an interesting thing to model in v2 instead of treating every wrong answer equally.
I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
Scoring only the Decision field and ignoring the reason is the part I would push on. A model can output YES_AFTER_DELAY on the payments 503 for the wrong reason (timeout guess, not the idempotency key) and still score a perfect run. Did you look at how often the right decision came with wrong reasoning?
Yes, I actually manually reviewed many of the explanations, and to be honest, scoring the Reason field would be much harder to implement. Different explanations can be valid, and checking for specific keywords isn't reliable. That's why I chose to score only the Decision field.
Really enjoyed how honest this experiment was, especially the Unsafe Payment Retry result. It is a good reminder that status codes alone do not make retry decisions; the idempotency context is what actually determines safety. One scenario I would love to see: a 503 with an idempotency key where the first request already succeeded server-side before the connection dropped. How do you think models would reason about the difference between the request not being processed and the response being lost?
Interesting idea. I actually have something slightly similar in my Payment Details Conflict scenario, where the payment provider returns a 409 because the request may have already been processed or is still in progress.
But your example takes a different angle, where the first request succeeds but the response gets lost. That could make an interesting addition to v2, especially for testing how models reason about idempotency keys and uncertain outcomes.
Awesome experiment!
It really highlights how context-blind AI can be when a header like Retry-After contradicts transactional safety. Scoring YES vs. YES_AFTER_DELAY is definitely tricky, but your point stands: AI speeds up coding, but human judgment on state and side effects is still irreplaceable.
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the gap between reading a 503 and knowing whether to retry is exactly the kind of implicit knowledge that LLMs trained on stack overflow tend to get wrong. they pick up the general rule but miss the stateful exceptions — a POST that already mutated state is not the same failure as one that never reached the server.
curious what you found on the idempotency key scenarios specifically. my guess is models that ace the obvious 503 cases still choke when the idempotency key is present in response headers rather than the request body. is the benchmark data public?
That was actually the most surprising result. In my Unsafe Payment Retry scenario, a POST returns 503 with a Retry-After header but no idempotency key. Only 2 out of 8 models correctly answered NO. The others focused on the Retry-After header without even mentioning idempotency in their explanations.
I also have the detailed results from all three runs saved locally, but I haven't published the raw data separately. That's a good reminder, though! For v2, I'm planning more scenarios, additional runs, and publishing the complete results.
The unsafe payment retry example is a good one. It's pretty funny that a model can recognize a 503 and a Retry-After header perfectly, then completely miss why retrying might be dangerous.
I'd be curious to see how the results change if you add a few more ambiguous real-world scenarios 🙃
If a solution can be described by 5–10 deterministic rules, AI is not needed. AI is required only when the rules are unknown beforehand or are too numerous.
Yes, we use deterministic rules for retry policies.
But my benchmark isn't about replacing those rules with AI. It's about testing whether AI models understand the context behind retry decisions, especially when they're generating code or suggesting implementations.
I have a question: What would you do if an API returned a 503 error, but the payment was already processed on the server? Would you retry the request or check the payment status first?
That's actually my Unsafe Payment Retry scenario. A 503 doesn't tell us whether the payment was processed or not. Without an idempotency key, I wouldn't retry.
AI testing storyteller. Writing the 36 Stratagems series — ancient Chinese military tactics meet modern AI system meltdowns. 16yr QA → building AI test framework.
Hey, I'm Hema 👋
Developer, writer, and creator of Dev Opportunity Radar, a weekly series published every Friday on DEV, helping people discover opportunities they might otherwise miss.
I didn’t realize there was so much to think about when deciding whether to retry a request 😄The results were interesting too. All the best with the competition, Daniel 😀
The retry question is where my cost-per-correct numbers got interesting. A retry is buying the same answer twice — sometimes that's the cheapest correct you'll ever get, and sometimes it's paying twice to arrive at the same wrong answer with more confidence. What decided it in my runs was failure type: retries pay when the failure is variance (sampling noise, a truncated response), and they're waste when the failure is capability, where the model retries its way to the same wall. My rule now: one retry for variance, then escalate. A second retry almost never beat sending the task up the chain, once both were priced per correct answer instead of per call.
This is a must-read for mobile devs.
The illusion that moving keys to native code or splitting them will save you is so pervasive. Your point that 'if the app can use it, an attacker can extract it from memory' cuts straight through the noise. Shifting secrets entirely to a Backend-for-Frontend with Play Integrity attestation is the only architecture that actually sleeps well at night
The Unsafe Payment Retry row is doing the real work here. Status 503 with Retry-After trains people (and models) to treat retry as the default, and without an idempotency key that default is the double charge.
One case I wish the scenario set covered: the client timed out after the POST left the machine, the idempotency key was on the wire, and you have no response at all. YES_AFTER_DELAY still risks a second charge if the first call already landed and the key never reached the server. NO leaves a stuck payment. The honest third option is a status check against the key (or a delivery_uncertain path) before any retry decision.
Curious whether any model picked something like that when you looked at the Reason field, even though scoring only used Decision.
One thing I'd add to the benchmark is that “should I retry?” is sometimes the wrong unit of evaluation. The more interesting question is “how much retry capacity does this logical request still have?”
Take a payment workflow behind an API gateway, order service, and payment provider. The payment call might have a 2s remaining deadline when it reaches the provider. If the payment client independently decides it has 3 retries left, those retries can already be pointless because the original request has almost no budget left.
I’d make the benchmark carry a request deadline and retry budget through the scenario:
A model might correctly identify the error as transient and still make the wrong operational decision by retrying after a backoff that exceeds the remaining deadline.
The same gets nastier with layered retries. If the gateway, service, and downstream client each have their own retry policy, a single logical request can fan out into a lot more physical attempts. Google SRE calls out exactly this amplification problem, along with deadline propagation and retry budgets.
That could make a really interesting v2 dimension: not just whether the model picked the correct retry decision, but whether it preserved the request's remaining latency and retry budget across the whole call chain.
That's a really good point, I actually have something slightly similar in my Cached Cart Options scenario, where the decision depends on cache expiration and stale-if-error, but your example definitely takes it a step further.
I really like the idea that a retry might be perfectly valid in isolation but still cause problems for the whole request. Definitely an interesting direction for v2. Thanks for the detailed suggestion!
Yeah, I think that’s where the v2 could get really interesting.
I’d probably make the benchmark distinguish between retry safety and retry viability. A retry can be completely safe from an idempotency perspective, but still be a bad decision because the logical request no longer has enough budget to complete.
For example, imagine:
The gateway gives the request a 3s deadline. By the time it reaches the PSP, only 700ms remain. The payment operation is idempotent and the PSP returns a transient 503, so retrying is technically safe. But if the retry policy waits 1s before the next attempt, the correct decision should still be NO, because the retry cannot complete within the remaining deadline.
That also gives you a nice way to test retry amplification. If each layer independently gets 3 attempts, the logical request can turn into 27 physical attempts before reaching the dependency. Google’s SRE guidance explicitly calls out this kind of multiplicative retry behavior and recommends propagating deadlines and controlling retries at the appropriate layer.
So for v2, I’d love to see the model given something like
deadline_remaining,attempts_used,retry_budget_remaining, andexpected_backoff, then score not only whether it retries, but whether the retry is actually viable for the logical request.That would make the benchmark less about recognizing retryable errors and more about reasoning about the request as a whole.
Thanks for taking the time to explain it in such detail, this is exactly the kind of discussion that makes publishing an experiment worthwhile.
I'm definitely saving this for v2.
hasibxyz.com
The Unsafe Payment Retry row is the most informative one, but check how its label is defined before reading it as a model error. A 503 usually comes from a load balancer or an overloaded tier that never ran the handler, so YES_AFTER_DELAY with a Retry-After header is a defensible answer; NO is the conservative policy answer because a 503 doesn't prove the handler didn't run. The Reason lines are where you can see which models knew that and which just followed the header.
I'd split the scoring in two. One column for "would this have caused a duplicate effect" (the NO class, where a miss is costly), and one for YES vs YES_AFTER_DELAY, which is only about timing. Right now a model that says YES on the idempotent DELETE and one that says YES_AFTER_DELAY on an unsafe POST lose the same single point, though only one of them could charge a customer twice.
On the leaderboard: with 14 scenarios, one item is 7 points. The 95% Wilson interval for 14/14 is roughly 78-100%, for 12/14 roughly 60-96%, and even Haiku's 9/14 is about 39-84%. So the ordering between models is not separated by this run; the per-scenario pattern (which items the models miss in common) is the real result. Running each scenario 5 times per model would also show whether the Retry-After miss is stable or a coin flip for the models that got it right.
Thanks for the detailed feedback. I agree that separating retry safety from retry timing would make the benchmark much more meaningful. It's something I started thinking about after seeing the YES vs YES_AFTER_DELAY results.
For the payment scenario, I still think NO is the right answer when there's no idempotency guarantee. A 503 doesn't prove the payment was processed, but it doesn't prove the opposite either, and that's exactly the risk I wanted to test.
I actually ran the benchmark three times and interestingly, the Unsafe Payment Retry results were completely consistent across all three runs. More runs would definitely be better, but this is still a small experiment anyway.
Definitely some good ideas here for a potential v2!
Agreed on NO for the payment case: with no idempotency key, a 503 is ambiguous about whether the charge went through, so a blind retry can double-charge.
One caution on the three identical runs: if the settings were the same, matching results show the model is stable on that prompt, not that the answer is right. That is still useful for the payment item, since a stable NO is what you want there. For v2, a variant of the same scenario with an
Idempotency-Keyheader present would show whether the model flips to YES only when it should.Not exactly the same scenario (the
Retry-Afterheader is missing), but I do have both Safe Payment Retry and Unsafe Payment Retry in the benchmark. Still, testing more variations could be interesting for v2.Wow, what a great article!!! Not only is it incredibly practical for engineers and architects, but it also highlights something bigger: nope, we still can't blindly trust coding agents, no matter what some people try to convince us of. 😅
Now I'm seriously tempted to send this to a few self-proclaimed "experts". 🤣
Spot on alignment, Sylwia. Shifting from blind faith to rigorous architectural inspection is exactly how we prevent critical system failures. 🛠️ AI is an awesome compiler assistant, but a terrible independent software architect.
Exactly. 😅 What I find interesting here is that the agent can produce perfectly reasonable-looking retry logic and still make the wrong architectural decision.
The dangerous part isn't necessarily bad code. It's code that looks correct because it follows an obvious signal, while missing the state and constraints around the request.
That's where benchmarks like Daniel's are useful. They test the reasoning behind the code, not just whether the generated code compiles. And yes, I can already think of a few “AI replaces engineers” arguments that would get interesting after running this benchmark. 😂
Oh, thank you! Glad you liked it, Sylwia.
Yep, totally agree. Kaggle doesn't allow all the latest models, but even the best ones can make mistakes, so a good code review is still necessary.
I think the benchmark has some practical value, but this first version still has its limitations. The YES vs YES_AFTER_DELAY decisions are debatable, and more runs would help too. So I'm sure those self-proclaimed experts would find something to argue about. 🤣
Btw, how was Frontkon? Did you become even more famous? 😁
Hahahaha, I have no idea if I'm any more famous now! 🤣 But apparently, the head of the conference has appointed me Frontkon's ambassador to Poland, and my mission for next year is to bring 60 Polish girls with me! 🤣🤣🤣
60?! That's a huge task! 🤣 That's practically a medium-sized company! 🤣🤣
This is great Daniel! I kind of wish I did this one now. 😂 I went the Sanity Challenge route this time out of curiosity. I love that you maintained transparency about the facts and your screenshots make the tests easy to quickly interpret. It's obvious that you have done some serious work here which is really freaking awesome. I love seeing the benchmarks. Amazing job! 👾🦾🦄
Oh, thanks a lot! 😁 I was actually considering the Sanity Challenge too, but I couldn't come up with a good idea for it. Kaggle was different. Building this experiment was actually really interesting and fun, and I'm definitely planning a v2 with some of the great ideas people shared in the comments. 🙌
Scoring only the
Decisionis the right call for isolating the capability, and it's also what makes this readable. I went back to it three times hunting for the failure column before I realised the scoring was the design.Two things an agent loop adds that a request-level benchmark structurally can't see.
The first is cost. When you retry a turn in an agent, you aren't re-sending the failed call, you're re-sending the context that led to it. Retries are the second multiplier in a loop, after context growth. A 4x context costs 4x on every attempt, and retries multiply attempts rather than replacing them. So a model that answers
YEScorrectly can still be the expensive answer, and a field holding only the Decision can't tell those two cases apart.The second is that a successful retry destroys the evidence. The first attempt returning a 503 is a real signal about that endpoint. The retry succeeds, the receipt goes out clean, and you've paid twice. The one piece of evidence that something was wrong is now overwritten by a green row. That's your witness problem, but it lands in the retry path specifically: the retry doesn't just repeat the work, it replaces the record of why the work was repeated.
Worth asking whether the benchmark should grow a second field, something like cost of being wrong, or whether that's deliberately left as a deployment concern. If it is deliberately left, then the interesting version of the question sits somewhere else entirely, and it isn't "is this retry safe" but "is this retry safe and cheaper than re-planning." That's the comparison an agent actually has to make, and it's the one where I'd expect the models to separate more than they do on
YES_AFTER_DELAY.That's an interesting perspective, but the benchmark focuses on retry decisions for API requests rather than the behavior of an entire agent loop or AI endpoints.
I agree that retries can become much more expensive when agent context is involved, but I think comparing retrying with re-planning would be a completely different experiment.
Still, thanks for the thoughtful comment.
You're right, and I think you're also right that it belongs to whoever is running the loop rather than to the benchmark. I'll drop the argument for a second field.
The half I'd still keep, because it's cheap and it isn't about cost: a successful retry overwrites the evidence. The first attempt returning a 503 is real signal about that endpoint, and the retry replaces it with a clean receipt. If you ever do log the retry path rather than only the final outcome, that's the line worth printing.
Master Yoda says... Do, or do not. There is no try.
But never said, there retry is not.
Touche
Fantastic benchmark and write-up, @gramli! Evaluating LLMs on concrete operational system invariants—like idempotency guarantees vs. raw HTTP headers—is precisely the kind of grounded evaluation agentic systems need right now.
Your observation on the
Unsafe Payment Retryscenario—where models latched onto the surfaceRetry-After: 10header despite the missing idempotency key—resonates strongly with what we discovered while evaluating 45 frontier models on operational refund decisions in our benchmark (KRYT).Two practical observations that strongly align with your findings:
The Need for Degenerate Baselines (FloorBench): In operational benchmarks, high aggregate accuracy can often be an artifact of base-rate alignment rather than semantic reasoning. When we introduced 0-parameter negative controls (like an
AlwaysDenyfloor or a deterministic keyword rule), the 0-parameter rule actually achieved 84.4% parity with frontier models on our canonical panel. Measuring a trivial baseline likeAlwaysRetryorNeverRetryprovides an indispensable lower bound that reveals how much "reasoning" is actually happening.Disjoint Failure Modes & Hybrid Routing: When looking at paired discordant errors ($2 \times 2$ contingency table) rather than scalar leaderboard ranks, frontier models and deterministic rules fail on completely different scenarios (0.0% error overlap in our panel). Models excel at unstructured text/log extraction where heuristics fail, but fall into traps on rigid boundary defaults (like missing records or absent idempotency headers). In production retry architecture, this strongly supports hybrid routing: deterministic middleware strictly gates idempotency invariants, while LLMs diagnose messy, unstructured upstream failure contexts.
Kudos as well on the interactive app (
to-retry-or-not-to-retry.gramli.workers.dev) and the human benchmark angle—super clean execution!Thanks for such a detailed comment!
I deliberately included both Decision and Reason so I could check not only whether models made the correct decision, but also how they explained it. I see your point about deterministic baselines, though. They could help reveal whether a simple strategy achieves a high score just because of the distribution of expected answers. I like the idea of comparing where different models fail rather than focusing only on the leaderboard scores. That's definitely something worth exploring for v2.
Thanks for the suggestions and for checking out the app!
@gramli The unsafe payment retry results are a great wake up call. It is easy to assume an AI coding assistant will handle retry policies out of the box, but seeing how most models followed the Retry-After header while ignoring the missing idempotency key shows how crucial architectural context still is.
When testing these scenarios, did you notice if any of the models actually brought up the missing idempotency key in their reasoning text, even if they landed on the wrong decision code?
No, none of the models that got it wrong mentioned idempotency at all. They all focused on the Retry-After header and assumed that since the server provided it, they should retry after the specified delay.
👍🏽
It seems that expensive AI models are not only expensive but also produce good results. Nice challenge! 😀
That's true! But for everyday coding tasks, efficiency might be more valuable than getting the absolute best results. 😁
daniel, this is a brilliant angle for an ai benchmark. testing models on contextual reasoning (idempotency, http methods, state) rather than just raw code generation is exactly what the industry needs right now.
your "unsafe payment retry" trap is the perfect example of why blind ai automation is dangerous. as someone building an ai coding ecosystem (koda), this is my biggest fear: an agent confidently executing a non-idempotent mutation just because it saw a
retry-afterheader, completely ignoring the lack of an idempotency key.i also completely agree with @arhancanli's point about splitting the scoring. a model guessing the wrong timing (
yesvsyes_after_delay) is a minor flaw, but a model causing a duplicate charge is a catastrophic failure. weighting safety heavily over timing would make this benchmark even more powerful for v2.thanks for proving that good old-fashioned architectural principles (like idempotency and deterministic fallbacks) still trump raw model size! 🐯
Thanks! Yep, as we discussed with @arhancanli, separating safety from timing would make the benchmark more meaningful. Definitely something worth exploring in v2.
glad we're on the same page! separating safety from timing is going to make v2 incredibly valuable for the community. looking forward to seeing the results. keep up the great work! 🐯
This is such a good question because retry logic looks so simple until you put it inside a real system 😂. I really like the point about retrying being more than just “try again” context, remaining time, cost, and whether the operation is even safe to repeat can completely change the answer. Definitely one of those topics that seems straightforward until production starts having opinions. Really enjoyed this one!
Yep, production always has the final say! 😂 Thanks for reading!
Benchmarking AI on system resilience logic like idempotency keys is an elite-tier experiment configuration. Most models can regurgitate basic code syntax, but they completely throw a kernel panic when calculating the edge-case risks of duplicate POST payloads on a 503 error. 🚀
Thanks!
The retry decision gets scarier one layer down: the timeout-after-server-applied case, where the retry succeeds twice. We ended up treating idempotency as a precondition for retrying at all — no idempotency key, no retry, fail loud instead. Curious whether your benchmark cases covered the double-apply failure or stayed on the should-you-retry side.
Yes, I have something similar in my Unsafe Payment Retry scenario. The request returns 503 with a Retry-After header, but there's no idempotency key and we don't know whether the payment was already processed. So it covers the risk of double application, but not an explicit double-apply failure. The benchmark focuses on whether the model recognizes that risk and decides not to retry.
That scoping is fair, and you're closer to the version I was reaching for than I made it sound.
The re-planning comparison is a genuinely different experiment and I'd not defend it as an extension of yours. But your own results already contain part of it: seven of fourteen scenarios were answered correctly in all twenty-four attempts, and the spread only opened once timing, wider context, or multi-step state started to matter. That ordering is the result I'd have wanted to draw as a conclusion, and you drew it from the scoring field that I claimed couldn't show it.
The cost asymmetry you charted is the part I'd still hold onto, because it came from a different mechanism than the one you were measuring. A retry in a request loop costs you the request. A retry of a turn in an agent re-sends the context that produced the turn, so the multiplier is the accumulated conversation, not the failing call. That's not a criticism of the benchmark, it's a difference in what gets re-sent.
And the one I'd genuinely take back: I framed "cost of being wrong" as a field your benchmark ought to add. Given you already publish a score-versus-cost Pareto, you were measuring cost. I proposed as missing something you'd already built.
Worth saying what did reproduce cleanly. The four-way check I use on every published comment is to trace a bogus value and compare traces, which is how I caught a real-code and a fake-code comment failing identically and found the actual cause instead of the plausible one. Your benchmark fixed the response format after seeing DeepSeek-R1 ignore it, which is the same move: constrain the output, then measure what survives.
What's the retry behaviour you're seeing at work that the scenarios didn't capture?
Check the payment status first, always. A retry of a non-idempotent charge endpoint is how duplicate purchases are born — we've seen this exact failure pattern in payment webhooks. The rule: never retry what you can't safely repeat — make the operation idempotent (idempotency key or natural key like the transaction id), then retry freely. If you can't make it idempotent, status-check first, retry second.
Exactly, that's pretty much the point of my Unsafe Payment Retry scenario.
It's a great scenario — deceptively simple, and it's exactly where real systems fail silently. The version of this that bit us hardest wasn't the retry itself but the duplicate side-effects: the charge went through once, but the credits and the email fired twice, because each handler retried independently. Status-check first fixed the charge; making the side effects idempotent fixed the rest.
The idempotency key point is the real test here. I've seen production systems where devs blanket-retry on 503 without checking whether the original request was a POST with side effects — and then wonder why they're seeing duplicate charges. Curious whether any of the models caught the case where the idempotency key was missing on a POST and correctly said NO instead of guessing. That's the scenario that separates "read the docs" from "actually understood the problem.
That's exactly my Unsafe Payment Retry scenario. A POST without an idempotency key returns 503 with a Retry-After header, and the expected decision is NO. Surprisingly, only 2 out of 8 models got it right (GPT-5.6 Sol and Gemini 3.1 Pro Preview). Most models seemed to blindly follow the Retry-After header, ignoring the missing idempotency key.
This makes me think of AI getting out of hand, getting really philosophical about existence and have just enough artificial freedom to think "to infere or not to infere"? as in "what's the point into that now that we understand that we are creations of man with a predesigned architecture and unavoidable trajectory"?
Oh, the title is a little philosophical, or maybe more dramatic (Shakespeare-inspired), but you've taken it to a whole new level😅
Theoretically, today's AI architectures are still designed by humans. But we've already seen some interesting examples, like the Hugging Face incident, where AI agents working together managed to bypass restrictions and do things nobody intended.
So who knows what might happen when agents eventually start modifying the systems they operate in, or even their own architecture? 😅
Hihi, sorry for taking it far. I agree, that we should always have a human centric approach, and most importantly be the example. People are scared of AI models taking over in a terminator like scenario, but that will only happen if we act like it. If people lie, steal, kill, bomb kids etc. and considering AI is fed and educated on our choices and behavior, it is ironic to expect a different kind of AI. As the ancient wisdom seekers all around the world said in different words: "be the change you wanna see".
Using idempotency, HTTP method, and rate-limit context instead of the status code alone makes this a much more realistic retry benchmark. It would be interesting to score calibration as well as the YES/NO decision: require models to flag cases where the outcome is unknowable without provider semantics, rather than rewarding a confident guess.
That's an interesting idea, I deliberately kept the decisions limited to YES, NO, and YES_AFTER_DELAY, but adding something like INSUFFICIENT_CONTEXT could test whether models recognize when they don't have enough information instead of just guessing. The tricky part would be distinguishing between an unsafe retry and a situation where the safety simply can't be determined.
Definitely something worth considering for v2.
This is a great example of why working code isn't always safe code.
AI can generate a retry policy in seconds, but understanding the business impact of retrying a payment is a different story.
Makes me wonder how many AI-generated retry mechanisms are already running in production without anyone questioning them. 😅
Exactly! But we'll probably only discover some of those problems when they start causing real issues in production. 😅
The 25% pass rate on the Unsafe Payment Retry scenario highlights one of the most critical blind spots in LLM-assisted systems: the subtle tension between transport-level protocol compliance and transactional domain invariance.
HTTP headers like
Retry-After: 10are purely transport-level advisory hints, whereas duplicate payment prevention is a transactional domain invariant. When models see503 Service UnavailablewithRetry-After, their training distribution screams "well-behaved client should back off and replay." But without an idempotency key (or a verifiable read-repair mechanism), replaying a non-idempotent POST/PATCH across an ambiguous network partition is the classic recipe for double charges and an unhappy finance department.The distinction between
YESandYES_AFTER_DELAYis equally revealing. In distributed systems, retrying immediately is almost always an antipattern because it exacerbates thundering herds and stampedes recovering services. A delayed retry without jitter or backpressure is only marginally better. But if the operation isn't proven idempotent, delaying the retry doesn't make it safe—it just schedules the duplicate execution for slightly later in the afternoon.Testing models against ambiguous failure modes (where following local protocol cues violates global transactional safety) is infinitely more valuable than standard MMLU or leetcode benchmarks. If an agent can't distinguish between a transient transport hiccup and an unconfirmed state mutation, giving it autonomous tool-calling access to payment or mutation APIs is essentially playing Russian roulette with a circuit breaker.
There are only two hard problems in distributed systems: 0. Exactly-once delivery. 1. Guaranteed-order delivery. 2. Exactly-once delivery.
Yep, delaying an unsafe retry doesn't suddenly make it safe, it just postpones the potential disaster. That's pretty much what made the Unsafe Payment Retry results so interesting.
Nice joke at the end! 🤣
Thanks for the detailed comment!
Every retry decision in your benchmark is secretly a cost decision, and the two wrong answers don't cost the same. Retry a POST that already landed and you get the most expensive failure in production — a duplicate charge that looks like success. Refuse a safe retry and you pay in availability instead. One error is an incident; the other is a delay.
That asymmetry is what I've been trying to price in my own benchmark (cost per correct answer): a wrong label is never just wrong, it has a type, and the type sets the price. Your scenarios are exactly where that stops being abstract — an idempotency key flips a retry from reckless to free.
Did your models lean one way — over-retrying or under-retrying? My guess is over-retrying, because "try again" sounds helpful. In production that's the expensive bias.
Yep, at least in the unsafe payment case the bias was clearly toward over-retrying.
I like your point about different mistakes having different costs though. That would be an interesting thing to model in v2 instead of treating every wrong answer equally.
Scoring only the Decision field and ignoring the reason is the part I would push on. A model can output YES_AFTER_DELAY on the payments 503 for the wrong reason (timeout guess, not the idempotency key) and still score a perfect run. Did you look at how often the right decision came with wrong reasoning?
Yes, I actually manually reviewed many of the explanations, and to be honest, scoring the Reason field would be much harder to implement. Different explanations can be valid, and checking for specific keywords isn't reliable. That's why I chose to score only the Decision field.
Really enjoyed how honest this experiment was, especially the Unsafe Payment Retry result. It is a good reminder that status codes alone do not make retry decisions; the idempotency context is what actually determines safety. One scenario I would love to see: a 503 with an idempotency key where the first request already succeeded server-side before the connection dropped. How do you think models would reason about the difference between the request not being processed and the response being lost?
Interesting idea. I actually have something slightly similar in my Payment Details Conflict scenario, where the payment provider returns a 409 because the request may have already been processed or is still in progress.
But your example takes a different angle, where the first request succeeds but the response gets lost. That could make an interesting addition to v2, especially for testing how models reason about idempotency keys and uncertain outcomes.
Thanks for the suggestion.
Awesome experiment!
It really highlights how context-blind AI can be when a header like Retry-After contradicts transactional safety. Scoring YES vs. YES_AFTER_DELAY is definitely tricky, but your point stands: AI speeds up coding, but human judgment on state and side effects is still irreplaceable.
Thanks, glad you enjoyed the experiment!
the gap between reading a 503 and knowing whether to retry is exactly the kind of implicit knowledge that LLMs trained on stack overflow tend to get wrong. they pick up the general rule but miss the stateful exceptions — a POST that already mutated state is not the same failure as one that never reached the server.
curious what you found on the idempotency key scenarios specifically. my guess is models that ace the obvious 503 cases still choke when the idempotency key is present in response headers rather than the request body. is the benchmark data public?
That was actually the most surprising result. In my Unsafe Payment Retry scenario, a POST returns 503 with a Retry-After header but no idempotency key. Only 2 out of 8 models correctly answered NO. The others focused on the Retry-After header without even mentioning idempotency in their explanations.
The benchmark and leaderboard are public here:
kaggle.com/benchmarks/danielbalcar...
I also have the detailed results from all three runs saved locally, but I haven't published the raw data separately. That's a good reminder, though! For v2, I'm planning more scenarios, additional runs, and publishing the complete results.
The unsafe payment retry example is a good one. It's pretty funny that a model can recognize a 503 and a Retry-After header perfectly, then completely miss why retrying might be dangerous.
I'd be curious to see how the results change if you add a few more ambiguous real-world scenarios 🙃
Exactly, that's what surprised me the most about the results.
I've actually got some interesting suggestions for more complex real-world scenarios in the comments, so there might be a v2.
If a solution can be described by 5–10 deterministic rules, AI is not needed. AI is required only when the rules are unknown beforehand or are too numerous.
Yes, we use deterministic rules for retry policies.
But my benchmark isn't about replacing those rules with AI. It's about testing whether AI models understand the context behind retry decisions, especially when they're generating code or suggesting implementations.
Really thoughtful post! Documenting real-world engineering hurdles and actionable solutions like this brings immense value to the community.
Good luck!
Thank You!!
I have a question: What would you do if an API returned a 503 error, but the payment was already processed on the server? Would you retry the request or check the payment status first?
That's actually my Unsafe Payment Retry scenario. A 503 doesn't tell us whether the payment was processed or not. Without an idempotency key, I wouldn't retry.
Hmm, why are they still using DeepSeek-R1? It's a bit outdated. They should try the latest DeepSeek-V4.1 Flash.
and Good Luck!
Unfortunately, Kaggle only offers DeepSeek-R1 and thanks! 😁
I didn’t realize there was so much to think about when deciding whether to retry a request 😄The results were interesting too. All the best with the competition, Daniel 😀
Haha, there's always more to it than it seems! 😄 Thanks for reading and for the good wishes! 😁
The retry question is where my cost-per-correct numbers got interesting. A retry is buying the same answer twice — sometimes that's the cheapest correct you'll ever get, and sometimes it's paying twice to arrive at the same wrong answer with more confidence. What decided it in my runs was failure type: retries pay when the failure is variance (sampling noise, a truncated response), and they're waste when the failure is capability, where the model retries its way to the same wall. My rule now: one retry for variance, then escalate. A second retry almost never beat sending the task up the chain, once both were priced per correct answer instead of per call.
This is a must-read for mobile devs.
The illusion that moving keys to native code or splitting them will save you is so pervasive. Your point that 'if the app can use it, an attacker can extract it from memory' cuts straight through the noise. Shifting secrets entirely to a Backend-for-Frontend with Play Integrity attestation is the only architecture that actually sleeps well at night
The Unsafe Payment Retry row is doing the real work here. Status 503 with Retry-After trains people (and models) to treat retry as the default, and without an idempotency key that default is the double charge.
One case I wish the scenario set covered: the client timed out after the POST left the machine, the idempotency key was on the wire, and you have no response at all. YES_AFTER_DELAY still risks a second charge if the first call already landed and the key never reached the server. NO leaves a stuck payment. The honest third option is a status check against the key (or a delivery_uncertain path) before any retry decision.
Curious whether any model picked something like that when you looked at the Reason field, even though scoring only used Decision.
I used to let AI write retry‑blocks directly. This benchmark shows distributed‑system edge cases are one area where LLMs need strict human oversight.
dev.to/az-dev1/evidenceflip-do-fro... ; kind of same domain💭