DEV Community

Cover image for To Retry or Not to Retry? That Is the Question.

To Retry or Not to Retry? That Is the Question.

Daniel Balcarek on October 08, 2026

This is a submission for the Kaggle Benchmarking Challenge Those who read my articles know that a lot of them are actually benchmarks of something...
Collapse
 
sinarezaei profile image
Sina Rezaei •

One thing I'd add to the benchmark is that “should I retry?” is sometimes the wrong unit of evaluation. The more interesting question is “how much retry capacity does this logical request still have?”

Take a payment workflow behind an API gateway, order service, and payment provider. The payment call might have a 2s remaining deadline when it reaches the provider. If the payment client independently decides it has 3 retries left, those retries can already be pointless because the original request has almost no budget left.

I’d make the benchmark carry a request deadline and retry budget through the scenario:

logical_request_deadline = T+2s
elapsed = 1.4s
remaining_budget = 600ms
attempts_used = 2
retryable = true
Enter fullscreen mode Exit fullscreen mode

A model might correctly identify the error as transient and still make the wrong operational decision by retrying after a backoff that exceeds the remaining deadline.

The same gets nastier with layered retries. If the gateway, service, and downstream client each have their own retry policy, a single logical request can fan out into a lot more physical attempts. Google SRE calls out exactly this amplification problem, along with deadline propagation and retry budgets.

That could make a really interesting v2 dimension: not just whether the model picked the correct retry decision, but whether it preserved the request's remaining latency and retry budget across the whole call chain.

Collapse
 
gramli profile image
Daniel Balcarek •

That's a really good point, I actually have something slightly similar in my Cached Cart Options scenario, where the decision depends on cache expiration and stale-if-error, but your example definitely takes it a step further.

I really like the idea that a retry might be perfectly valid in isolation but still cause problems for the whole request. Definitely an interesting direction for v2. Thanks for the detailed suggestion!

Collapse
 
sinarezaei profile image
Sina Rezaei •

Yeah, I think that’s where the v2 could get really interesting.

I’d probably make the benchmark distinguish between retry safety and retry viability. A retry can be completely safe from an idempotency perspective, but still be a bad decision because the logical request no longer has enough budget to complete.

For example, imagine:

API Gateway
  -> Order Service
      -> Payment Service
          -> PSP
Enter fullscreen mode Exit fullscreen mode

The gateway gives the request a 3s deadline. By the time it reaches the PSP, only 700ms remain. The payment operation is idempotent and the PSP returns a transient 503, so retrying is technically safe. But if the retry policy waits 1s before the next attempt, the correct decision should still be NO, because the retry cannot complete within the remaining deadline.

That also gives you a nice way to test retry amplification. If each layer independently gets 3 attempts, the logical request can turn into 27 physical attempts before reaching the dependency. Google’s SRE guidance explicitly calls out this kind of multiplicative retry behavior and recommends propagating deadlines and controlling retries at the appropriate layer.

So for v2, I’d love to see the model given something like deadline_remaining, attempts_used, retry_budget_remaining, and expected_backoff, then score not only whether it retries, but whether the retry is actually viable for the logical request.

That would make the benchmark less about recognizing retryable errors and more about reasoning about the request as a whole.

Thread Thread
 
gramli profile image
Daniel Balcarek •

Thanks for taking the time to explain it in such detail, this is exactly the kind of discussion that makes publishing an experiment worthwhile.

I'm definitely saving this for v2.

Thread Thread
 
hasibul0909 profile image
Comment marked as low quality/non-constructive by the community. View Code of Conduct
MD HASIB •

hasibxyz.com

Collapse
 
arhancanli profile image
Arhan Canli •

The Unsafe Payment Retry row is the most informative one, but check how its label is defined before reading it as a model error. A 503 usually comes from a load balancer or an overloaded tier that never ran the handler, so YES_AFTER_DELAY with a Retry-After header is a defensible answer; NO is the conservative policy answer because a 503 doesn't prove the handler didn't run. The Reason lines are where you can see which models knew that and which just followed the header.

I'd split the scoring in two. One column for "would this have caused a duplicate effect" (the NO class, where a miss is costly), and one for YES vs YES_AFTER_DELAY, which is only about timing. Right now a model that says YES on the idempotent DELETE and one that says YES_AFTER_DELAY on an unsafe POST lose the same single point, though only one of them could charge a customer twice.

On the leaderboard: with 14 scenarios, one item is 7 points. The 95% Wilson interval for 14/14 is roughly 78-100%, for 12/14 roughly 60-96%, and even Haiku's 9/14 is about 39-84%. So the ordering between models is not separated by this run; the per-scenario pattern (which items the models miss in common) is the real result. Running each scenario 5 times per model would also show whether the Retry-After miss is stable or a coin flip for the models that got it right.

Collapse
 
gramli profile image
Daniel Balcarek •

Thanks for the detailed feedback. I agree that separating retry safety from retry timing would make the benchmark much more meaningful. It's something I started thinking about after seeing the YES vs YES_AFTER_DELAY results.

For the payment scenario, I still think NO is the right answer when there's no idempotency guarantee. A 503 doesn't prove the payment was processed, but it doesn't prove the opposite either, and that's exactly the risk I wanted to test.

I actually ran the benchmark three times and interestingly, the Unsafe Payment Retry results were completely consistent across all three runs. More runs would definitely be better, but this is still a small experiment anyway.

Definitely some good ideas here for a potential v2!

Collapse
 
arhancanli profile image
Arhan Canli •

Agreed on NO for the payment case: with no idempotency key, a 503 is ambiguous about whether the charge went through, so a blind retry can double-charge.

One caution on the three identical runs: if the settings were the same, matching results show the model is stable on that prompt, not that the answer is right. That is still useful for the payment item, since a stable NO is what you want there. For v2, a variant of the same scenario with an Idempotency-Key header present would show whether the model flips to YES only when it should.

Thread Thread
 
gramli profile image
Daniel Balcarek •

Not exactly the same scenario (the Retry-After header is missing), but I do have both Safe Payment Retry and Unsafe Payment Retry in the benchmark. Still, testing more variations could be interesting for v2.

Collapse
 
sylwia-lask profile image
Sylwia Laskowska •

Wow, what a great article!!! Not only is it incredibly practical for engineers and architects, but it also highlights something bigger: nope, we still can't blindly trust coding agents, no matter what some people try to convince us of. 😅

Now I'm seriously tempted to send this to a few self-proclaimed "experts". 🤣

Collapse
 
alexcodebytes profile image
Oleksandr •

Spot on alignment, Sylwia. Shifting from blind faith to rigorous architectural inspection is exactly how we prevent critical system failures. 🛠️ AI is an awesome compiler assistant, but a terrible independent software architect.

Collapse
 
sinarezaei profile image
Sina Rezaei •

Exactly. 😅 What I find interesting here is that the agent can produce perfectly reasonable-looking retry logic and still make the wrong architectural decision.

The dangerous part isn't necessarily bad code. It's code that looks correct because it follows an obvious signal, while missing the state and constraints around the request.

That's where benchmarks like Daniel's are useful. They test the reasoning behind the code, not just whether the generated code compiles. And yes, I can already think of a few “AI replaces engineers” arguments that would get interesting after running this benchmark. 😂

Collapse
 
gramli profile image
Daniel Balcarek •

Oh, thank you! Glad you liked it, Sylwia.

Yep, totally agree. Kaggle doesn't allow all the latest models, but even the best ones can make mistakes, so a good code review is still necessary.

I think the benchmark has some practical value, but this first version still has its limitations. The YES vs YES_AFTER_DELAY decisions are debatable, and more runs would help too. So I'm sure those self-proclaimed experts would find something to argue about. 🤣

Btw, how was Frontkon? Did you become even more famous? 😁

Collapse
 
sylwia-lask profile image
Sylwia Laskowska •

Hahahaha, I have no idea if I'm any more famous now! 🤣 But apparently, the head of the conference has appointed me Frontkon's ambassador to Poland, and my mission for next year is to bring 60 Polish girls with me! 🤣🤣🤣

Thread Thread
 
gramli profile image
Daniel Balcarek •

60?! That's a huge task! 🤣 That's practically a medium-sized company! 🤣🤣

Collapse
 
annavi11arrea1 profile image
Anna Villarreal •

This is great Daniel! I kind of wish I did this one now. 😂 I went the Sanity Challenge route this time out of curiosity. I love that you maintained transparency about the facts and your screenshots make the tests easy to quickly interpret. It's obvious that you have done some serious work here which is really freaking awesome. I love seeing the benchmarks. Amazing job! 👾🦾🦄

Collapse
 
gramli profile image
Daniel Balcarek •

Oh, thanks a lot! 😁 I was actually considering the Sanity Challenge too, but I couldn't come up with a good idea for it. Kaggle was different. Building this experiment was actually really interesting and fun, and I'm definitely planning a v2 with some of the great ideas people shared in the comments. 🙌

Collapse
 
theagentloop profile image
The Agent Loop •

Scoring only the Decision is the right call for isolating the capability, and it's also what makes this readable. I went back to it three times hunting for the failure column before I realised the scoring was the design.

Two things an agent loop adds that a request-level benchmark structurally can't see.

The first is cost. When you retry a turn in an agent, you aren't re-sending the failed call, you're re-sending the context that led to it. Retries are the second multiplier in a loop, after context growth. A 4x context costs 4x on every attempt, and retries multiply attempts rather than replacing them. So a model that answers YES correctly can still be the expensive answer, and a field holding only the Decision can't tell those two cases apart.

The second is that a successful retry destroys the evidence. The first attempt returning a 503 is a real signal about that endpoint. The retry succeeds, the receipt goes out clean, and you've paid twice. The one piece of evidence that something was wrong is now overwritten by a green row. That's your witness problem, but it lands in the retry path specifically: the retry doesn't just repeat the work, it replaces the record of why the work was repeated.

Worth asking whether the benchmark should grow a second field, something like cost of being wrong, or whether that's deliberately left as a deployment concern. If it is deliberately left, then the interesting version of the question sits somewhere else entirely, and it isn't "is this retry safe" but "is this retry safe and cheaper than re-planning." That's the comparison an agent actually has to make, and it's the one where I'd expect the models to separate more than they do on YES_AFTER_DELAY.

Collapse
 
gramli profile image
Daniel Balcarek •

That's an interesting perspective, but the benchmark focuses on retry decisions for API requests rather than the behavior of an entire agent loop or AI endpoints.

I agree that retries can become much more expensive when agent context is involved, but I think comparing retrying with re-planning would be a completely different experiment.

Still, thanks for the thoughtful comment.

Collapse
 
theagentloop profile image
The Agent Loop •

You're right, and I think you're also right that it belongs to whoever is running the loop rather than to the benchmark. I'll drop the argument for a second field.

The half I'd still keep, because it's cheap and it isn't about cost: a successful retry overwrites the evidence. The first attempt returning a 503 is real signal about that endpoint, and the retry replaces it with a clean receipt. If you ever do log the retry path rather than only the final outcome, that's the line worth printing.

Collapse
 
bryan_hamilton profile image
Bryan Hamilton •

Master Yoda says... Do, or do not. There is no try.

Collapse
 
gramli profile image
Daniel Balcarek •

But never said, there retry is not.

Collapse
 
bryan_hamilton profile image
Bryan Hamilton •

Touche

Collapse
 
vaibhav_lalwani2241351_b profile image
VAIBHAV LALWANI 2241351 •

Fantastic benchmark and write-up, @gramli! Evaluating LLMs on concrete operational system invariants—like idempotency guarantees vs. raw HTTP headers—is precisely the kind of grounded evaluation agentic systems need right now.

Your observation on the Unsafe Payment Retry scenario—where models latched onto the surface Retry-After: 10 header despite the missing idempotency key—resonates strongly with what we discovered while evaluating 45 frontier models on operational refund decisions in our benchmark (KRYT).

Two practical observations that strongly align with your findings:

  1. The Need for Degenerate Baselines (FloorBench): In operational benchmarks, high aggregate accuracy can often be an artifact of base-rate alignment rather than semantic reasoning. When we introduced 0-parameter negative controls (like an AlwaysDeny floor or a deterministic keyword rule), the 0-parameter rule actually achieved 84.4% parity with frontier models on our canonical panel. Measuring a trivial baseline like AlwaysRetry or NeverRetry provides an indispensable lower bound that reveals how much "reasoning" is actually happening.

  2. Disjoint Failure Modes & Hybrid Routing: When looking at paired discordant errors ($2 \times 2$ contingency table) rather than scalar leaderboard ranks, frontier models and deterministic rules fail on completely different scenarios (0.0% error overlap in our panel). Models excel at unstructured text/log extraction where heuristics fail, but fall into traps on rigid boundary defaults (like missing records or absent idempotency headers). In production retry architecture, this strongly supports hybrid routing: deterministic middleware strictly gates idempotency invariants, while LLMs diagnose messy, unstructured upstream failure contexts.

Kudos as well on the interactive app (to-retry-or-not-to-retry.gramli.workers.dev) and the human benchmark angle—super clean execution!

Collapse
 
gramli profile image
Daniel Balcarek •

Thanks for such a detailed comment!

I deliberately included both Decision and Reason so I could check not only whether models made the correct decision, but also how they explained it. I see your point about deterministic baselines, though. They could help reveal whether a simple strategy achieves a high score just because of the distribution of expected answers. I like the idea of comparing where different models fail rather than focusing only on the leaderboard scores. That's definitely something worth exploring for v2.

Thanks for the suggestions and for checking out the app!

Collapse
 
sushyam_nagallapati profile image
Sushyam Nagallapati •

@gramli The unsafe payment retry results are a great wake up call. It is easy to assume an AI coding assistant will handle retry policies out of the box, but seeing how most models followed the Retry-After header while ignoring the missing idempotency key shows how crucial architectural context still is.

When testing these scenarios, did you notice if any of the models actually brought up the missing idempotency key in their reasoning text, even if they landed on the wrong decision code?

Collapse
 
gramli profile image
Daniel Balcarek •

No, none of the models that got it wrong mentioned idempotency at all. They all focused on the Retry-After header and assumed that since the server provided it, they should retry after the specified delay.

Collapse
 
sushyam_nagallapati profile image
Sushyam Nagallapati •

👍🏽

Collapse
 
webdeveloperhyper profile image
Web Developer Hyper •

It seems that expensive AI models are not only expensive but also produce good results. Nice challenge! 😀

Collapse
 
gramli profile image
Daniel Balcarek •

That's true! But for everyday coding tasks, efficiency might be more valuable than getting the absolute best results. 😁

Collapse
 
koda2026 profile image
Harun - solo dev •

daniel, this is a brilliant angle for an ai benchmark. testing models on contextual reasoning (idempotency, http methods, state) rather than just raw code generation is exactly what the industry needs right now.

your "unsafe payment retry" trap is the perfect example of why blind ai automation is dangerous. as someone building an ai coding ecosystem (koda), this is my biggest fear: an agent confidently executing a non-idempotent mutation just because it saw a retry-after header, completely ignoring the lack of an idempotency key.

i also completely agree with @arhancanli's point about splitting the scoring. a model guessing the wrong timing (yes vs yes_after_delay) is a minor flaw, but a model causing a duplicate charge is a catastrophic failure. weighting safety heavily over timing would make this benchmark even more powerful for v2.

thanks for proving that good old-fashioned architectural principles (like idempotency and deterministic fallbacks) still trump raw model size! 🐯

Collapse
 
gramli profile image
Daniel Balcarek •

Thanks! Yep, as we discussed with @arhancanli, separating safety from timing would make the benchmark more meaningful. Definitely something worth exploring in v2.

Collapse
 
koda2026 profile image
Harun - solo dev •

glad we're on the same page! separating safety from timing is going to make v2 incredibly valuable for the community. looking forward to seeing the results. keep up the great work! 🐯

Collapse
 
khaithisran profile image
thisran •

This is such a good question because retry logic looks so simple until you put it inside a real system 😂. I really like the point about retrying being more than just “try again” context, remaining time, cost, and whether the operation is even safe to repeat can completely change the answer. Definitely one of those topics that seems straightforward until production starts having opinions. Really enjoyed this one!

Collapse
 
gramli profile image
Daniel Balcarek •

Yep, production always has the final say! 😂 Thanks for reading!

Collapse
 
alexcodebytes profile image
Oleksandr •

Benchmarking AI on system resilience logic like idempotency keys is an elite-tier experiment configuration. Most models can regurgitate basic code syntax, but they completely throw a kernel panic when calculating the edge-case risks of duplicate POST payloads on a 503 error. 🚀

Collapse
 
gramli profile image
Daniel Balcarek •

Thanks!

Collapse
 
mrsaynothing profile image
Mr Say Nothing •

The retry decision gets scarier one layer down: the timeout-after-server-applied case, where the retry succeeds twice. We ended up treating idempotency as a precondition for retrying at all — no idempotency key, no retry, fail loud instead. Curious whether your benchmark cases covered the double-apply failure or stayed on the should-you-retry side.

Collapse
 
gramli profile image
Daniel Balcarek •

Yes, I have something similar in my Unsafe Payment Retry scenario. The request returns 503 with a Retry-After header, but there's no idempotency key and we don't know whether the payment was already processed. So it covers the risk of double application, but not an explicit double-apply failure. The benchmark focuses on whether the model recognizes that risk and decides not to retry.

Collapse
 
theagentloop profile image
The Agent Loop •

That scoping is fair, and you're closer to the version I was reaching for than I made it sound.

The re-planning comparison is a genuinely different experiment and I'd not defend it as an extension of yours. But your own results already contain part of it: seven of fourteen scenarios were answered correctly in all twenty-four attempts, and the spread only opened once timing, wider context, or multi-step state started to matter. That ordering is the result I'd have wanted to draw as a conclusion, and you drew it from the scoring field that I claimed couldn't show it.

The cost asymmetry you charted is the part I'd still hold onto, because it came from a different mechanism than the one you were measuring. A retry in a request loop costs you the request. A retry of a turn in an agent re-sends the context that produced the turn, so the multiplier is the accumulated conversation, not the failing call. That's not a criticism of the benchmark, it's a difference in what gets re-sent.

And the one I'd genuinely take back: I framed "cost of being wrong" as a field your benchmark ought to add. Given you already publish a score-versus-cost Pareto, you were measuring cost. I proposed as missing something you'd already built.

Worth saying what did reproduce cleanly. The four-way check I use on every published comment is to trace a bogus value and compare traces, which is how I caught a real-code and a fake-code comment failing identically and found the actual cause instead of the plausible one. Your benchmark fixed the response format after seeing DeepSeek-R1 ignore it, which is the same move: constrain the output, then measure what survives.

What's the retry behaviour you're seeing at work that the scenarios didn't capture?

Collapse
 
bpecha14 profile image
Bogdan •

Check the payment status first, always. A retry of a non-idempotent charge endpoint is how duplicate purchases are born — we've seen this exact failure pattern in payment webhooks. The rule: never retry what you can't safely repeat — make the operation idempotent (idempotency key or natural key like the transaction id), then retry freely. If you can't make it idempotent, status-check first, retry second.

Collapse
 
gramli profile image
Daniel Balcarek •

Exactly, that's pretty much the point of my Unsafe Payment Retry scenario.

Collapse
 
bpecha14 profile image
Bogdan •

It's a great scenario — deceptively simple, and it's exactly where real systems fail silently. The version of this that bit us hardest wasn't the retry itself but the duplicate side-effects: the charge went through once, but the credits and the email fired twice, because each handler retried independently. Status-check first fixed the charge; making the side effects idempotent fixed the rest.

Collapse
 
onizuka profile image
Onizuka •

The idempotency key point is the real test here. I've seen production systems where devs blanket-retry on 503 without checking whether the original request was a POST with side effects — and then wonder why they're seeing duplicate charges. Curious whether any of the models caught the case where the idempotency key was missing on a POST and correctly said NO instead of guessing. That's the scenario that separates "read the docs" from "actually understood the problem.

Collapse
 
gramli profile image
Daniel Balcarek •

That's exactly my Unsafe Payment Retry scenario. A POST without an idempotency key returns 503 with a Retry-After header, and the expected decision is NO. Surprisingly, only 2 out of 8 models got it right (GPT-5.6 Sol and Gemini 3.1 Pro Preview). Most models seemed to blindly follow the Retry-After header, ignoring the missing idempotency key.

Collapse
 
rinrinrinrin profile image
Rin Katsuragi •

This makes me think of AI getting out of hand, getting really philosophical about existence and have just enough artificial freedom to think "to infere or not to infere"? as in "what's the point into that now that we understand that we are creations of man with a predesigned architecture and unavoidable trajectory"?

Collapse
 
gramli profile image
Daniel Balcarek •

Oh, the title is a little philosophical, or maybe more dramatic (Shakespeare-inspired), but you've taken it to a whole new level😅

Theoretically, today's AI architectures are still designed by humans. But we've already seen some interesting examples, like the Hugging Face incident, where AI agents working together managed to bypass restrictions and do things nobody intended.

So who knows what might happen when agents eventually start modifying the systems they operate in, or even their own architecture? 😅

Collapse
 
rinrinrinrin profile image
Rin Katsuragi •

Hihi, sorry for taking it far. I agree, that we should always have a human centric approach, and most importantly be the example. People are scared of AI models taking over in a terminator like scenario, but that will only happen if we act like it. If people lie, steal, kill, bomb kids etc. and considering AI is fed and educated on our choices and behavior, it is ironic to expect a different kind of AI. As the ancient wisdom seekers all around the world said in different words: "be the change you wanna see".

Collapse
 
alexshev profile image
Alex Shev •

Using idempotency, HTTP method, and rate-limit context instead of the status code alone makes this a much more realistic retry benchmark. It would be interesting to score calibration as well as the YES/NO decision: require models to flag cases where the outcome is unknowable without provider semantics, rather than rewarding a confident guess.

Collapse
 
gramli profile image
Daniel Balcarek •

That's an interesting idea, I deliberately kept the decisions limited to YES, NO, and YES_AFTER_DELAY, but adding something like INSUFFICIENT_CONTEXT could test whether models recognize when they don't have enough information instead of just guessing. The tricky part would be distinguishing between an unsafe retry and a situation where the safety simply can't be determined.

Definitely something worth considering for v2.

Collapse
 
hassan95eb profile image
Hassan Amini •

This is a great example of why working code isn't always safe code.

AI can generate a retry policy in seconds, but understanding the business impact of retrying a payment is a different story.

Makes me wonder how many AI-generated retry mechanisms are already running in production without anyone questioning them. 😅

Collapse
 
gramli profile image
Daniel Balcarek •

Exactly! But we'll probably only discover some of those problems when they start causing real issues in production. 😅

Collapse
 
fgrandon profile image
Francisco Grandón •

The 25% pass rate on the Unsafe Payment Retry scenario highlights one of the most critical blind spots in LLM-assisted systems: the subtle tension between transport-level protocol compliance and transactional domain invariance.

HTTP headers like Retry-After: 10 are purely transport-level advisory hints, whereas duplicate payment prevention is a transactional domain invariant. When models see 503 Service Unavailable with Retry-After, their training distribution screams "well-behaved client should back off and replay." But without an idempotency key (or a verifiable read-repair mechanism), replaying a non-idempotent POST/PATCH across an ambiguous network partition is the classic recipe for double charges and an unhappy finance department.

The distinction between YES and YES_AFTER_DELAY is equally revealing. In distributed systems, retrying immediately is almost always an antipattern because it exacerbates thundering herds and stampedes recovering services. A delayed retry without jitter or backpressure is only marginally better. But if the operation isn't proven idempotent, delaying the retry doesn't make it safe—it just schedules the duplicate execution for slightly later in the afternoon.

Testing models against ambiguous failure modes (where following local protocol cues violates global transactional safety) is infinitely more valuable than standard MMLU or leetcode benchmarks. If an agent can't distinguish between a transient transport hiccup and an unconfirmed state mutation, giving it autonomous tool-calling access to payment or mutation APIs is essentially playing Russian roulette with a circuit breaker.

There are only two hard problems in distributed systems: 0. Exactly-once delivery. 1. Guaranteed-order delivery. 2. Exactly-once delivery.

Collapse
 
gramli profile image
Daniel Balcarek •

Yep, delaying an unsafe retry doesn't suddenly make it safe, it just postpones the potential disaster. That's pretty much what made the Unsafe Payment Retry results so interesting.

Nice joke at the end! 🤣

Thanks for the detailed comment!

Collapse
 
karmendra_pandey_43ac6983 profile image
karmendra pandey •

Every retry decision in your benchmark is secretly a cost decision, and the two wrong answers don't cost the same. Retry a POST that already landed and you get the most expensive failure in production — a duplicate charge that looks like success. Refuse a safe retry and you pay in availability instead. One error is an incident; the other is a delay.

That asymmetry is what I've been trying to price in my own benchmark (cost per correct answer): a wrong label is never just wrong, it has a type, and the type sets the price. Your scenarios are exactly where that stops being abstract — an idempotency key flips a retry from reckless to free.

Did your models lean one way — over-retrying or under-retrying? My guess is over-retrying, because "try again" sounds helpful. In production that's the expensive bias.

Collapse
 
gramli profile image
Daniel Balcarek •

Yep, at least in the unsafe payment case the bias was clearly toward over-retrying.

I like your point about different mistakes having different costs though. That would be an interesting thing to model in v2 instead of treating every wrong answer equally.

Collapse
 
kartik-nvjk profile image
Kartik N V J K •

Scoring only the Decision field and ignoring the reason is the part I would push on. A model can output YES_AFTER_DELAY on the payments 503 for the wrong reason (timeout guess, not the idempotency key) and still score a perfect run. Did you look at how often the right decision came with wrong reasoning?

Collapse
 
gramli profile image
Daniel Balcarek •

Yes, I actually manually reviewed many of the explanations, and to be honest, scoring the Reason field would be much harder to implement. Different explanations can be valid, and checking for specific keywords isn't reliable. That's why I chose to score only the Decision field.

Collapse
 
sgaggjhkjh profile image
Arjun Sharma •

Really enjoyed how honest this experiment was, especially the Unsafe Payment Retry result. It is a good reminder that status codes alone do not make retry decisions; the idempotency context is what actually determines safety. One scenario I would love to see: a 503 with an idempotency key where the first request already succeeded server-side before the connection dropped. How do you think models would reason about the difference between the request not being processed and the response being lost?

Collapse
 
gramli profile image
Daniel Balcarek •

Interesting idea. I actually have something slightly similar in my Payment Details Conflict scenario, where the payment provider returns a 409 because the request may have already been processed or is still in progress.

But your example takes a different angle, where the first request succeeds but the response gets lost. That could make an interesting addition to v2, especially for testing how models reason about idempotency keys and uncertain outcomes.

Thanks for the suggestion.

Collapse
 
dharu_g profile image
Dharu •

Awesome experiment!
It really highlights how context-blind AI can be when a header like Retry-After contradicts transactional safety. Scoring YES vs. YES_AFTER_DELAY is definitely tricky, but your point stands: AI speeds up coding, but human judgment on state and side effects is still irreplaceable.

Collapse
 
gramli profile image
Daniel Balcarek •

Thanks, glad you enjoyed the experiment!

Collapse
 
mudassirworks profile image
Mudassir Khan •

the gap between reading a 503 and knowing whether to retry is exactly the kind of implicit knowledge that LLMs trained on stack overflow tend to get wrong. they pick up the general rule but miss the stateful exceptions — a POST that already mutated state is not the same failure as one that never reached the server.

curious what you found on the idempotency key scenarios specifically. my guess is models that ace the obvious 503 cases still choke when the idempotency key is present in response headers rather than the request body. is the benchmark data public?

Collapse
 
gramli profile image
Daniel Balcarek •

That was actually the most surprising result. In my Unsafe Payment Retry scenario, a POST returns 503 with a Retry-After header but no idempotency key. Only 2 out of 8 models correctly answered NO. The others focused on the Retry-After header without even mentioning idempotency in their explanations.

The benchmark and leaderboard are public here:
kaggle.com/benchmarks/danielbalcar...

I also have the detailed results from all three runs saved locally, but I haven't published the raw data separately. That's a good reminder, though! For v2, I'm planning more scenarios, additional runs, and publishing the complete results.

Collapse
 
alkaznodemaven profile image
Alexandr Kazmin •

The unsafe payment retry example is a good one. It's pretty funny that a model can recognize a 503 and a Retry-After header perfectly, then completely miss why retrying might be dangerous.

I'd be curious to see how the results change if you add a few more ambiguous real-world scenarios 🙃

Collapse
 
gramli profile image
Daniel Balcarek •

Exactly, that's what surprised me the most about the results.

I've actually got some interesting suggestions for more complex real-world scenarios in the comments, so there might be a v2.

Collapse
 
ukrguru profile image
Oleksandr Viktor •

If a solution can be described by 5–10 deterministic rules, AI is not needed. AI is required only when the rules are unknown beforehand or are too numerous.

Collapse
 
gramli profile image
Daniel Balcarek •

Yes, we use deterministic rules for retry policies.

But my benchmark isn't about replacing those rules with AI. It's about testing whether AI models understand the context behind retry decisions, especially when they're generating code or suggesting implementations.

Collapse
 
dev_in_the_fog profile image
Jason Y. (dev_in_the_fog) •

Really thoughtful post! Documenting real-world engineering hurdles and actionable solutions like this brings immense value to the community.

Collapse
 
technogamerz profile image
𝐓𝐡𝐞 𝐋𝐚𝐳𝐲 𝐆𝐢𝐫𝐥 •

Good luck!

Collapse
 
gramli profile image
Daniel Balcarek •

Thank You!!

Collapse
 
rahul_r15 profile image
Rahul R •

I have a question: What would you do if an API returned a 503 error, but the payment was already processed on the server? Would you retry the request or check the payment status first?

Collapse
 
gramli profile image
Daniel Balcarek •

That's actually my Unsafe Payment Retry scenario. A 503 doesn't tell us whether the payment was processed or not. Without an idempotency key, I wouldn't retry.

Collapse
 
xulingfeng profile image
xulingfeng •

Hmm, why are they still using DeepSeek-R1? It's a bit outdated. They should try the latest DeepSeek-V4.1 Flash.
and Good Luck!

Collapse
 
gramli profile image
Daniel Balcarek •

Unfortunately, Kaggle only offers DeepSeek-R1 and thanks! 😁

Collapse
 
hemapriya_kanagala profile image
Hemapriya Kanagala •

I didn’t realize there was so much to think about when deciding whether to retry a request 😄The results were interesting too. All the best with the competition, Daniel 😀

Collapse
 
gramli profile image
Daniel Balcarek •

Haha, there's always more to it than it seems! 😄 Thanks for reading and for the good wishes! 😁

Collapse
 
karmendra_pandey_43ac6983 profile image
karmendra pandey •

The retry question is where my cost-per-correct numbers got interesting. A retry is buying the same answer twice — sometimes that's the cheapest correct you'll ever get, and sometimes it's paying twice to arrive at the same wrong answer with more confidence. What decided it in my runs was failure type: retries pay when the failure is variance (sampling noise, a truncated response), and they're waste when the failure is capability, where the model retries its way to the same wall. My rule now: one retry for variance, then escalate. A second retry almost never beat sending the task up the chain, once both were priced per correct answer instead of per call.

Collapse
 
dharu_g profile image
Dharu •

This is a must-read for mobile devs.
The illusion that moving keys to native code or splitting them will save you is so pervasive. Your point that 'if the app can use it, an attacker can extract it from memory' cuts straight through the noise. Shifting secrets entirely to a Backend-for-Frontend with Play Integrity attestation is the only architecture that actually sleeps well at night

Collapse
 
chrissellers profile image
Chris Sellers •

The Unsafe Payment Retry row is doing the real work here. Status 503 with Retry-After trains people (and models) to treat retry as the default, and without an idempotency key that default is the double charge.

One case I wish the scenario set covered: the client timed out after the POST left the machine, the idempotency key was on the wire, and you have no response at all. YES_AFTER_DELAY still risks a second charge if the first call already landed and the key never reached the server. NO leaves a stuck payment. The honest third option is a status check against the key (or a delivery_uncertain path) before any retry decision.

Curious whether any model picked something like that when you looked at the Reason field, even though scoring only used Decision.

Collapse
 
pixquill profile image
PixQuill •

I used to let AI write retry‑blocks directly. This benchmark shows distributed‑system edge cases are one area where LLMs need strict human oversight.

Collapse
 
az-dev1 profile image
Akash •

dev.to/az-dev1/evidenceflip-do-fro... ; kind of same domain💭