DEV Community

Rate Limits Are a Queueing Problem: Capacity Math for Agent Fleets

Debashish Ghosal on November 07, 2026

Rate Limits Are a Queueing Problem: Capacity Math for Agent Fleets Standup ends. Five people go back to their desks and invoke five agen...
Collapse
 
gautamtalksdev profile image
Gautam K •

Framing the standup burst as the arrival rate is the useful shift, because the daily average can sit well under the limit while a 90 second spike blows past it. Have you compared staggering agent starts with a small random delay against a shared queue with a concurrency cap? I'd guess the cap is easier to reason about, since the five hour window then becomes a budget the team can plan against.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

Yes, we see the bursts. Let me explain a bit - our agents run certain workloads and this involves EC2. The Linux EC2, no issues because aws scales well dynamically. However, our Mac runners we don’t. AWS charges for Mac runner for 24hrs regardless of usage, so the runner pool is fixed. Since runner pool is fixed, this is where the burst of agents hurt as now many agents are competing for resources where pool is fixed. As we are in 4 different time zones, people have adapted to kicking off their jobs as per TZ. Although standup happens at a fixed time for all TZs with best overlap. We encouraged folks not to kick off agents after standup as that’s when bursts happen. It’s not the best solution but it’s working with the constraints we have.

Collapse
 
gautamtalksdev profile image
Gautam K •

A fixed Mac pool is where the queueing math bites hardest, since there is no elastic capacity to absorb a burst. Asking people not to start agents right after standup is a fair fix that spreads arrivals without any new tooling. A small admission queue in front of the Mac runners would make the same rule automatic, so a burst waits in line instead of everyone fighting for the same hosts. Do the Mac jobs have deadlines, or could most of them run in the quieter hours of one of the time zones?

Thread Thread
 
debashish_ghosal profile image
Debashish Ghosal •

Quiet hours covers most of it — that 24-hour Mac billing stings precisely because the pool is sized for the burst, not the average, so anything that can wait, should. But there's always an interactive slice that can't wait, and that's what the admission queue is for: a small FIFO capped at pool size turns the 9:02 stampede into a line without asking anyone to remember a social rule at 9:02. So both, split by job type — queue for interactive, schedule for batch. The queue also hands you the number the article begs for: how long things actually waited today.

Thread Thread
 
gautamtalksdev profile image
Gautam K •

Splitting by job type is a clean answer: the queue protects the interactive slice, and the schedule keeps batch work out of the burst entirely. Daily wait time is the number I'd chart too, because it tells you when the pool is genuinely too small rather than just busy right after standup. Thanks for walking through the real constraints.

Thread Thread
 
debashish_ghosal profile image
Debashish Ghosal •

I think the wait chart might be the single most useful artifact to come out of that thread, because it turns the argument into a number you can argue with. Daily wait time separates the two diagnoses that look identical when you are staring at a spinner: a pool that is genuinely too small all day, and a pool that is merely herd-trampled for ninety minutes after standup. Split by job type is how I would wire it too, queue for the interactive slice, schedule for the batch, and the queue hands you the wait number for free. Threads that end in a concrete design like yours are the reason I end articles with a real question instead of "what do you think."

Collapse
 
delayrun profile image
delay.run •

Framing it as arrival rate vs service rate clears up so much. When five agents fire at 9:02, a bigger plan just makes the same spike a bit shorter, while a small jittered start window or a token bucket in front of the fleet fixes the shape of the problem for free. I'd be curious whether you saw retries make it worse: a rate-limited request that retries right away is basically a second arrival landing in the same busy minute.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

Yes — and worse than the naive math suggests. A retry is a second arrival in the same busy minute, and the part that really hurt was the synchronized wakeup: five agents hit 429 at the same second, all back off with the same exponent, all wake together and stampede again. The two fixes that earned their keep: honor Retry-After when the provider sends it, and count retries against the same token bucket as fresh calls, so a retrying agent competes with itself instead of with the fleet. Your "a bigger plan makes the same spike shorter" reading is exactly right — capacity changes the wait; the bucket and the jitter change the shape.

Collapse
 
reidmarlow profile image
Reid Marlow •

The hidden multiplier on the retry storm is that provider rate limits track rolling tokens rather than raw requests. When five agents get 429'd at the same second with twenty thousand tokens in flight, standard exponential backoff with full jitter still tends to synchronize. They all wake up right as the sixty-second token bucket refills, dump one hundred thousand tokens into the API at once, and knock each other back into rate limiting.

In my own multi-agent setups, what finally flattened the curve was decoupling tool execution time from LLM turns through a shared local leaky bucket. When one agent spends forty seconds waiting on a test suite or git clone, its slot on the token regulator opens up for another worker. Tracking token grants at the transport layer before the HTTP call ever fires keeps local concurrency high without ever triggering the provider backoff loop.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

You describe a really interesting scenario where exponential back off is a great idea but given same software logic is running and the burst of retries keep happening at specific intervals although the cohort count may change. I know one may suggest a complex coordinator etc to allocate “come back at this time”. I dealt with this a decade or so ago with constraints on some resource (pre AI) with a back off where the retry was a randomized number between two limits. It was not perfect but the bursts of retries coming back at the same time with lull in between went down. Quite cheap to implement and it worked on production workload. To be precise, this was for replaying dead letter queue on a Microsoft Exchange server where emails were not really real time like a slack message. I got away with that design. The bursts were constrained by the Netapp storage IO which is where the Exchange dumped the dead letter queue emails. Will this work do you think?

Collapse
 
reidmarlow profile image
Reid Marlow •

The ten-fold token gap between the council and the critique subagent explains why in-context debates rot.

When an LLM writes both sides of an eight-persona debate in a single prompt, it spends tokens simulating conversational tension. The personas disagree on phrasing while sharing the same generation priors and context assumptions. None of them inspect the filesystem or test whether a path exists.

The critique subagent ran heavier because fresh context with an adversarial brief spent those 130k tokens exploring system state and running shell checks. Paying for an independent check that drops the wrong rate from 31 percent to zero is a trade I would take on any production migration.

Collapse
 
debashish_ghosal profile image
Debashish Ghosal •

"The personas disagree on phrasing while sharing the same generation priors" — that's the precise reason my eight-persona council was theater: it had the appearance of independent review and none of the substance, because everything lived in one context that could talk itself into anything. Your numbers make the trade explicit: 130k tokens of fresh adversarial context is real money until you price the 31%-wrong rate it retired. An independent check with its own eyes beats a debate where one mind plays all the parts.