DEV Community

Cover image for How to Stop OAuth Token Stampedes in Zoho Integration Services (Python and Redis)
Sanya Mittal
Sanya Mittal

Posted on

How to Stop OAuth Token Stampedes in Zoho Integration Services (Python and Redis)

Eight workers share one Zoho refresh token. At minute 60 the cached access token expires, every worker gets a 401, and all eight request a new token at once. Retries pile on, Zoho throttles the refresh endpoint, and the queue stalls. This pattern shows up in order-sync and lead-sync jobs, and it is the first thing we check when we review how Zoho Integration Services work in production. Below, we build a single-flight token refresh in Python with Redis so that exactly one worker refreshes while the rest wait.

Context and Setup

Scenario: several Python workers (Celery, RQ, or FastAPI background tasks) push ERP or storefront data into Zoho CRM through the REST API, all using one OAuth client and one refresh token.

Prerequisites:

Python 3.10+, requests, and redis-py
Redis 6+ reachable from every worker
A Zoho refresh token with the scopes your calls need
The accounts URL for your data center

The accounts URL matters because the accounts server URL is specific to the data center where the user is located. Using accounts.zoho.com against an Indian or European org fails.

The limits are the real constraint. An access token is valid for one hour. A refresh token can create up to ten access tokens in ten minutes, and creation is blocked for the rest of that window once the limit is hit. Eight workers retrying a failed refresh burn through ten tokens in seconds.

Building Token-Safe Zoho Integration Services with Single-Flight Refresh

Single-flight means concurrent callers share one in-progress operation instead of each starting their own. The three steps below apply it to token refresh.

Step 1: Share One Token Across Zoho Integration Services Workers

Per-process token caches are the root cause, because each process refreshes on its own clock. Store the token in Redis with a TTL set slightly below Zoho’s expiry. Every worker then reads the same value, and Redis evicts it before it can go stale.

The flow each worker follows:

Read the token from Redis.
On a hit, call Zoho.
On a miss, try to take the refresh lock.
Lock winner refreshes and writes the token. Everyone else polls Redis.

Step 2: Add a Lock So Only One Worker Refreshes
python
import time
import uuid
import redis
import requests

r = redis.Redis(decode_responses=True)

ACCOUNTS = "https://accounts.zoho.com" # Why: must match your org's data center (.eu, .in, ...)
TOKEN_KEY, LOCK_KEY = "zoho:access_token", "zoho:refresh_lock"
SAFETY_MARGIN = 300 # Why: expire our copy 5 min early so no worker sends a dying token

Why: delete the lock only if we still own it (avoids freeing another worker's lock)

RELEASE = """
if redis.call('get', KEYS[1]) == ARGV[1] then
return redis.call('del', KEYS[1])
end
return 0
"""

def _refresh(cfg):
resp = requests.post(f"{ACCOUNTS}/oauth/v2/token", data={
"grant_type": "refresh_token",
"client_id": cfg["client_id"],
"client_secret": cfg["client_secret"],
"refresh_token": cfg["refresh_token"],
}, timeout=10)
body = resp.json()
if "access_token" not in body:
raise RuntimeError(f"Zoho refresh failed: {body}") # Why: fail loudly, never retry in a tight loop
r.set(TOKEN_KEY, body["access_token"], ex=body["expires_in"] - SAFETY_MARGIN)
return body["access_token"]

def get_token(cfg, wait=10):
deadline = time.time() + wait
while time.time() < deadline:
token = r.get(TOKEN_KEY)
if token:
return token
owner = str(uuid.uuid4())
# Why: SET NX is atomic, so exactly one worker wins under high concurrency
if r.set(LOCK_KEY, owner, nx=True, ex=15):
try:
return _refresh(cfg)
finally:
r.eval(RELEASE, 1, LOCK_KEY, owner)
time.sleep(0.2) # Why: losers poll the cache instead of hitting Zoho
raise TimeoutError("No Zoho token available within wait window")
Step 3: Retry Once on 401 and Weigh the Trade-offs

A token can still die early if someone revokes it or the clock drifts. Handle that with one bounded retry.

python
def zoho_get(cfg, path, **params):
for _ in range(2): # Why: one retry only, so a bad scope cannot loop forever
token = get_token(cfg)
resp = requests.get(
f"{cfg['api_domain']}{path}", params=params, timeout=15,
headers={"Authorization": f"Zoho-oauthtoken {token}"},
)
if resp.status_code != 401:
resp.raise_for_status()
return resp.json() if resp.content else {} # Why: Zoho returns 204 for empty searches
# Why: evict only if the cache still holds the rejected token,
# otherwise we would delete a fresh token another worker just wrote
if r.get(TOKEN_KEY) == token:
r.delete(TOKEN_KEY)
raise RuntimeError("Zoho rejected a fresh token; check scopes or revocation")

Trade-offs we weighed:

Redis lock vs. per-process refresh: the lock adds one dependency but removes the burst entirely.
Refresh on 401 vs. scheduled refresh: a scheduled job needs its own failure handling. Refresh-on-miss with a safety margin needs none.
Database row lock vs. Redis: a database works, but you must write TTL cleanup yourself.
Lock TTL: 15 seconds covers a slow accounts server. A shorter TTL risks two refreshes, and a longer one stalls workers if the winner crashes.

Real-World Application

In one of our Zoho Integration services projects at Oodles, a Celery worker pool syncing ERP orders into Zoho CRM suffered refresh bursts at token expiry that triggered Access Denied and left a stated number of jobs failing. We moved token storage to Redis with a single-flight lock and a bounded 401 retry.

Key Takeaways
Zoho issues at most ten access tokens per refresh token every ten minutes, so per-worker caches fail under concurrency.
Store one token in Redis with a TTL below Zoho’s one-hour expiry.
Use an atomic SET NX lock so one worker refreshes and the others poll the cache.
On a 401, evict the cached token only if it still matches the rejected one.
Retry once, then fail with a clear error, so scope and revocation problems surface fast.

Have a different Zoho failure mode, such as webhook channel expiry or bulk write errors? Drop it in the comments, or reach out about Zoho Integration services for your stack.

How long is a Zoho access token valid?
In Zoho Integration Services an access token is valid for one hour (3600 seconds). Refresh tokens do not expire until a user revokes them. Cache the access token centrally and renew it shortly before expiry instead of on every call.

How many access tokens can one refresh token generate?
Up to ten in a ten-minute span, after which creation is blocked for the rest of the window. In multi-worker deployments, share one cached token so refreshes stay far below that ceiling.

Why does Zoho return “Access Denied” or “invalid_code”?
In Zoho Integration Services Access Denied usually means you hit the token throttle. An invalid_code error appears when you generate an access token from a revoked refresh token. Also confirm your accounts URL matches the org’s data center.

Do Zoho Integration services need region-specific URLs?
In Zoho Integration Services Yes. Both the accounts server and the API domain depend on the org’s data center. Read api_domain from the token response rather than hardcoding it, and keep the accounts URL in configuration.

Should I use Redis or a database for the token cache?
Redis fits best because TTL expiry and atomic SET NX locking come built in. A database works if you add row locking and cleanup. Either way, only one refresh should run at a time.

Top comments (0)