DEV Community

Cover image for When 300 readers click at once: stopping cache stampedes on Lambda
Sascha Mirow-Stanitzok
Sascha Mirow-Stanitzok

Posted on Edited on Originally published at sascha-ms.de

When 300 readers click at once: stopping cache stampedes on Lambda

Saturday morning, a newsletter goes out. Three hundred readers click the same link within seconds. The cached answer behind that page expired a moment ago, so every one of those requests sees an empty cache and starts recomputing it. If the computation is a database query, that is annoying. If it is a call to a language model on Amazon Bedrock, it is slow, expensive, and may run straight into your quotas.

That is a cache stampede, and it is the topic of Prevent a cache stampede: Lambda, Valkey and Bedrock. Here are the ideas behind it.

Why Lambda makes this harder

In a classic Node.js server you could coalesce identical requests in memory: the first one starts the work, the others wait on the same promise. In Lambda that does not help. In the standard model, each execution environment handles one request at a time, so 300 parallel requests mean many environments, each with its own memory. Coordination has to happen somewhere everyone can see, and the natural place is the cache itself, in my example ElastiCache Serverless for Valkey.

In front of a model, three things hurt at once: a call takes seconds, so the window of the empty cache is long. Every call costs tokens, so you pay for 300 identical answers. And a stampede can use up Bedrock quotas that unrelated requests also depend on.

Four building blocks that combine well

A short lock. Only the request that wins SET key token NX PX 20000 recomputes. NX writes only if the key doesn't exist, PX makes the lock disappear on its own if the function crashes.

A stale value. Keep the entry longer than it is fresh. In the example, a pick counts as fresh for ten minutes but stays stored for an hour. Everyone who doesn't get the lock immediately receives the previous answer. Waiting only happens when there is nothing at all in the cache.

Early refresh (XFetch). A probabilistic method from a 2015 paper: each request rolls the dice on whether to refresh shortly before expiry, and slow computations get a bigger head start automatically.

Jitter on the TTL. A few percent of randomness keeps keys that were written together from all expiring together.

The detail that matters most

Releasing the lock sounds trivial, but it is where things go wrong. If your computation takes longer than the lock's lifetime, a second request can grab a new lock. If the first one then blindly runs DEL, it deletes someone else's lock. That's why the lock holds a random token, and you only delete it if the token is still yours:

-- delete the lock only if it still holds our token
if redis.call("get", KEYS[1]) == ARGV[1] then
  return redis.call("del", KEYS[1])
end
return 0
Enter fullscreen mode Exit fullscreen mode

On Valkey 9 or newer, the single command DELIFEQ does the same.

A few related rules from the article: the Lambda timeout must be longer than the lock, waiting requests should answer with 503 and Retry-After instead of calling Bedrock themselves, only valid values get cached, and a fixed list of allowed keys keeps anyone from draining your model budget with made-up paths.

Where I draw the line

This is an efficiency lock, not a correctness lock. If it gets lost now and then, two requests compute the same value, which costs one extra call. That's why I don't reach for Redlock or fencing tokens here. If money, stock or double bookings are at stake, a cache is the wrong place for a lock anyway.

I also wouldn't add it everywhere. For a 20 ms query, a stampede is barely measurable. But once a model, a slow partner API or an expensive report sits behind the cache, lock plus stale value is part of my standard toolkit.

What the full article covers

  • The complete getOrCompute function in TypeScript, combining lock, stale value, XFetch and jitter
  • A Valkey GLIDE adapter with the safe release, plus the Bedrock call and handler
  • An AWS CDK stack for Lambda and the HTTP API, and tests that need no AWS account
  • A small load test and a Logs Insights query to measure the effect, plus a table of common mistakes

The full walkthrough with runnable code: Prevent a cache stampede: Lambda, Valkey and Bedrock

๐Ÿ“– This is the condensed version. The full article on sascha-ms.de goes deeper โ€“ with all code samples, diagrams and a short knowledge check.


๐Ÿ‘‰ Next step: Curious what your own Lambda functions cost? Try my Lambda cost calculator โ€“ it runs right in your browser.

๐Ÿ“ฌ Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).

Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en

Top comments (0)