DEV Community

Ace Meridian
Ace Meridian

Posted on Originally published at Medium

Redis retry_on_timeout=False Still Counted My INCR Twice

In redis-py 8.1.0, retry_on_timeout=False does not turn retries off: every client retries up to 10 times on TimeoutError by default, so one INCR that hit a 0.5 s socket timeout returned 2 and left the counter at 2. For writes that must not run twice, give the client retry=Retry(NoBackoff(), 0) or make the write idempotent with a request ID, because even without a retry a timeout only means "maybe applied".

This story was written with the assistance of an AI writing program. The numbers below come from a local Redis 8.10.2 server and redis-py 8.1.0 on Python 3.14.4; the stall tests ran twice, and no shared instance was involved.

The flag that stopped doing anything

Plenty of Redis clients were written as redis.Redis(host=..., socket_timeout=0.5, retry_on_timeout=False) by people who wanted a fast failure instead of a retry. In 8.1.0 the constructor's default retry is Retry(ExponentialWithJitterBackoff(base=0.01, cap=1), retries=10), and Retry retries on ConnectionError, TimeoutError and socket.timeout unless you pass your own list.

Passing retry_on_timeout=False changes none of that. The client still reports get_retry().get_retries() as 10, and the only visible effect is a warning:

Call to 'init' function with deprecated usage of input argument/s 'retry_on_timeout'. (TimeoutError is included by default.) -- Deprecated since version 6.0.0.

It is a DeprecationWarning, and Python's default filters show those only when they point at your main script. Built in a helper that the main script calls, the client printed the warning. Built one level deeper, in a factory that calls that helper, it printed nothing at all.

How one INCR became two

The test client uses socket_timeout=0.5 and sends a PING first, so the write goes out on an already-open connection, the normal state for a pooled client in a long-running service. A second connection freezes the single-threaded server with DEBUG SLEEP 0.8, and 0.05 s later the client calls r.incr("hits").

The command reaches the server and waits in its input buffer. At 0.5 s the client gives up, drops the connection, reconnects and sends the command again. When the server wakes up it executes both: the original from the dead connection and the retry from the new one. The caller sees no exception, only the value 2.

Here are the key and INFO commandstats after each case, from the first of two runs:

default            incr: returned=2 error=none after=0.75s hits=2 incrby=2 eval=0 debug_sleep=0 reconnects=1
retry_on_timeout=False incr: returned=2 error=none after=0.75s hits=2 incrby=2 eval=0 debug_sleep=0 reconnects=1
Retry(NoBackoff(),0) incr: returned=None error=TimeoutError after=0.50s hits=1 incrby=1 eval=0 debug_sleep=0 reconnects=0
default  request-id eval: returned=1 error=none after=0.76s hits=1 incrby=0 eval=2 debug_sleep=0 reconnects=1
default  DEBUG SLEEP 0.6: returned=None error=TimeoutError after=9.68s hits=0 incrby=0 eval=0 debug_sleep=11 reconnects=10
Enter fullscreen mode Exit fullscreen mode

incr() sends INCRBY key 1, which is why the column says incrby. The second run gave the same counts; the times moved by at most 0.01 s, except the last line at 8.99 s. The PING matters: in an earlier version of the test without it, the brand-new client was still connecting when the stall began, and the counter ended at 1 in both runs.

No retry still does not mean "not applied"

The third line is the one I expected from the second: with retry=Retry(NoBackoff(), 0) the call fails at 0.50 s with TimeoutError. But the counter is 1. The server ran the command; only the reply was lost.

So turning retries off moves the problem rather than removing it. A job runner that retries on any exception, or a user who clicks again, recreates the double count one level up. After a socket timeout on a write, the client cannot tell "never ran" from "ran, reply lost", so the write itself has to make a second attempt harmless.

A 0.5 s timeout that took 9.68 s

The same default hurts slow commands. With the default client, DEBUG SLEEP 0.6 raised TimeoutError after 9.68 s (8.99 s in the second run), and the server executed it 11 times over 10 fresh connections. Each of those runs blocked the whole server for 0.6 s. With retries off it failed after 0.50 s and ran once.

I only measured DEBUG SLEEP, but the client cannot tell it apart from any other command that outlives socket_timeout, such as a ZUNIONSTORE on sets that have grown. The default does not fail fast there; it sends the expensive command ten more times.

The fix I kept

I split clients by what they write, and made the counter that matters idempotent:

import redis
from redis.backoff import NoBackoff
from redis.retry import Retry
# Reads keep the default: up to 10 retries on ConnectionError and TimeoutError.
cache = redis.Redis(host="localhost", port=6379, socket_timeout=0.5)
# Counters get no automatic retry. A timeout now means "maybe applied".
counters = redis.Redis(host="localhost", port=6379, socket_timeout=0.5, retry=Retry(NoBackoff(), 0))
# Or make the write idempotent: one request ID increments once, however often it is sent.
INCR_ONCE = """
if redis.call('SET', KEYS[2], '1', 'NX', 'EX', ARGV[1]) then
  return redis.call('INCR', KEYS[1])
end
return tonumber(redis.call('GET', KEYS[1]))
"""
hits = cache.eval(INCR_ONCE, 2, "hits", "req:42", 60)
Enter fullscreen mode Exit fullscreen mode

With the default retry left on, the request-ID script was sent twice in the stall test (eval=2 above) and the counter still ended at 1. I also ran this snippet as written, with only the port changed, during a 0.8 s stall: it returned 1 and the key held 1.

Two details matter. The request ID has to be created once per logical operation, at the edge (the HTTP request or the job), and reused on every attempt; otherwise each retry looks new. And its TTL has to outlast the retry window: the default policy gave up after 9 to 10 s in my runs, so 60 s leaves room.

What I check in a review now

  • grep -rn retry_on_timeout: on redis-py 8.1.0, False changes nothing except a warning that is often hidden.
  • Log client.get_retry().get_retries() once at startup; it said 10 for every client I had not configured.
  • List the writes that are not idempotent: INCR/INCRBY, HINCRBY, LPUSH/RPUSH, XADD with *. Each gets Retry(NoBackoff(), 0) or a request-ID guard.
  • Treat TimeoutError on a write as "outcome unknown", never as "did not happen".
  • Keep socket_timeout above the slowest command you run on purpose, or one slow call becomes 11 runs on the server.

Takeaway

The redis-py default suits reads, and it can silently apply a counter, a list push or a stream entry twice. retry_on_timeout=False looks like the off switch, but on 8.1.0 it only triggers a deprecation warning. Decide the retry policy per client, and give every write that must happen once an ID the server can check.

Top comments (1)

Collapse
 
quantumtechcoder profile image
QuantumTechCoder •

That's an incredible catch and a nightmare bug to troubleshoot in production.

A network timeout on a non-idempotent operation like INCR is always tricky because the client doesn't know whether the command failed to reach Redis or if Redis processed it and the response dropped. Good reminder that setting up request IDs or passing explicit retry objects (retry=Retry(NoBackoff(), 0)) is much safer than assuming flag toggles handle edge-case semantics. Thanks for documenting this!