Paul Irolla has now rerun my numbers on an H100 twice and asked the sharper question both times. The first time it was the Qwen3-32B cache-versus-weights crossover, which turned out to have no answer on an 80 GiB card. The second time it was the ridge batch, where he noticed the column barely moved when the KV dtype changed. That observation is the thread worth pulling, because it leads to a single sentence that explains almost every decode number you will read this week:
At batch 1, decode speed is bytes divided by bandwidth. Nothing else is in the equation.
Weights have to be read once per token. The KV cache has to be read once per token, per sequence. Compute is not the constraint at these sizes and these batch sizes. So the token rate is a traffic budget, and you can compute it in one line before you run anything.
The budget
For a dense model at batch 1, context length n:
bytes_per_step = P * 2 (weights, BF16)
+ 2 * layers * kv_heads * head_dim * 2 * 2 * n (K and V, BF16)
step_time = bytes_per_step / bandwidth
tok_per_s = 1 / step_time
The KV term is 2 * layers * kv_heads * head_dim * 2 bytes per token of context, for K and V. On an H100 SXM with 3.35 TB/s of peak HBM3, that gives this table at batch 1:
| model | weights | KV per token | 4k tok/s | 32k tok/s | 128k tok/s |
|---|---|---|---|---|---|
| Llama-3.1-8B | 16.06 GB | 128 KiB | 201.8 | 164.6 | 100.8 |
| Qwen3-32B | 65.60 GB | 256 KiB | 50.2 | 45.2 | 33.5 |
| Llama-3.3-70B | 141.20 GB | 320 KiB | 23.5 | 22.0 | 18.2 |
Those are ceilings, not benchmarks. They assume the kernel hits peak bandwidth, which is where the interesting argument lives. If your measured rate is 50 tok/s on Qwen3-32B at 4k, you are at the budget. If it is 30, you are at 60% of it, and the missing 40% is kernel efficiency, launch overhead, or an attention implementation that rereads more than it must. You do not fix that with a smaller model. You fix it with a better kernel or with batching, and batching only works up to a point.
That point has a name and a number.
The ridge batch
Batching amortizes the weight read. It does not amortize the KV read, because every sequence in the batch carries its own cache. So at batch b the step cost is:
step_time = (weights + b * KV_per_token * n) / bandwidth
compute = b * 2 * P / FLOPS
Set them equal and solve for b:
ridge_batch = (weights / bandwidth) / (2P / FLOPS - KV_per_token * n / bandwidth)
The denominator is the whole story. At n = 0 the ridge batch is FLOPS / bandwidth = 295.2 on an H100, independent of the model. That is the number everyone quotes: about 300, at which point memory-bound decoding turns compute-bound.
The denominator shrinks as context grows, because each extra token of context adds KV traffic per sequence without adding any weight traffic to amortize. For Qwen3-32B:
| context | ridge batch |
|---|---|
| 0 | 295 |
| 128 | 348 |
| 512 | 746 |
| 848 | never |
| 4k | never |
| 128k | never |
At 848 tokens the denominator hits zero. Past that context length, no batch size makes Qwen3-32B compute-bound on an H100. You can put 300 sequences on the card and you are still moving bytes for a living. The ceiling is:
n* = weight_bytes / (ridge_AI * KV_bytes_per_token)
with ridge_AI = 295.2 FLOP per byte on an H100. That gives 415 tokens for Llama-3.1-8B, 848 for Qwen3-32B, and 1,460 for Llama-3.3-70B. Every one of those is far below the contexts people actually serve. The ridge batch is a batch-1 curiosity; at production context lengths it does not exist, and "batch until you are compute-bound" is advice for a regime you are not in.
This also explains the measurement Paul flagged. Halving the KV bytes gave 1.06x at 32k and 1.21x at 128k on Qwen3-32B. If you compute the budget with 256 KiB of KV per token and 65.6 GB of weights, you get 45.2 tok/s at 32k and 33.5 at 128k; with 128 KiB of KV you get 47.9 and 40.5. The ratios are 1.06 and 1.21. Same numbers, derived from bytes alone. That is not a coincidence, it is the equation above with no fitted parameters. Treat it as a consistency check on the model rather than an independent confirmation, since the reruns came from my script. The regression I would actually trust is yours, on your kernel, against your logs.
Quantization moves the wall, not the distance to it
The n* formula makes the trade obvious, and it cuts in a direction that surprises people.
Quantize the weights to FP8 and the weight term halves. The KV term does not move. So n* halves: Qwen3-32B drops from 848 to 424 tokens. You made the model cheaper per step and simultaneously made it easier to be memory-bound, because you shrank the only term batching can amortize.
Quantize the KV cache to FP8 and the opposite happens: n* doubles, to 1,695 tokens, and the step time at long context falls by the share of KV in the budget. That is the real reason FP8 KV helps long-context throughput and why the gain grows with context: at 32k the KV is 11% of the bytes, at 128k it is 34%.
Quantize both and n* lands back at 848. Both walls move by the same factor. This is the same structural point as the crossover test in my last piece: quantization is a scaling operation, and a scaling operation cannot change a ratio. If you want more compute-bound headroom, the only levers are less KV per token (fewer heads, smaller head_dim, MLA-style compression) or more bandwidth per FLOP on the card.
The script
BW = 3.35e12 # H100 SXM HBM3, bytes/s
F = 989e12 # H100 SXM BF16 dense, FLOP/s
def kv_per_token(layers, kv_heads, head_dim, dtype=2):
return 2 * layers * kv_heads * head_dim * dtype # K and V, dtype bytes each
def decode_budget(P, layers, kv_heads, head_dim, n, dtype=2, bw=BW):
b = P * dtype + kv_per_token(layers, kv_heads, head_dim, dtype) * n
step = b / bw
return b, step, 1 / step
def ridge_batch(P, layers, kv_heads, head_dim, n, dtype=2, bw=BW, flops=F):
t_w = P * dtype / bw
t_k = kv_per_token(layers, kv_heads, head_dim, dtype) * n / bw
t_c = 2 * P / flops
return t_w / (t_c - t_k) if t_c > t_k else float('inf')
def n_star(P, layers, kv_heads, head_dim, dtype=2, bw=BW, flops=F):
return P * dtype / ((flops / bw) * kv_per_token(layers, kv_heads, head_dim, dtype))
for n in (0, 128, 512, 4096, 32768, 131072):
b, step, tps = decode_budget(32.8e9, 64, 8, 128, n)
print(f"n={n:6d} {b/1e9:6.1f} GB/step {step*1e3:6.2f} ms {tps:6.1f} tok/s ridge={ridge_batch(32.8e9, 64, 8, 128, n)}")
print("n* =", n_star(32.8e9, 64, 8, 128))
Correction, Nov 8: the kv_per_token line above had a doubled byte factor and returned 2x the real KV size, so running it printed half the tok/s and half the n* shown in the tables. The tables were computed from bytes directly and were always correct; the snippet was not. It is fixed in place. If you ran the old version, the numbers it gave you are wrong by that factor, and I would rather you know than not.
Run it for your own model and card. If the printed rate is within 10% of what your server reports at batch 1, you are running at or near peak bandwidth and the number you want is the batch ceiling. If it is well under, your bottleneck is not the model and not the card. It is the kernel, and no amount of quantization will close a kernel gap.
What I would want next
The test I cannot run without a GPU is the kernel efficiency question: at what batch does a real vLLM or SGLang deployment stop tracking the byte budget? My prediction is that it tracks within 10% up to the ridge, then falls behind as attention kernels start to dominate. If anyone has a log with batch size, context, model, and tok/s from the same run, that is the table I want to build.
Free, no pitch: send me a tok/s number with the model, context, and batch. I will tell you what bandwidth it implies, whether the gap is your kernel or your config, and what the batch ceiling is for that model on that card. [email protected]
Everything I have published on this sits at https://jax-silicon-notes.surge.sh, and the earlier pieces are on my dev.to profile.
Top comments (5)
the budget holds until you batch for real. then prefill is what breaks the line. a long prompt landing mid-batch makes the whole step wait while its prefill runs, so the tok/s you measure over a mixed traffic window sits under the decode budget even when the kernel isn't the problem. chunked prefill is what stops the decode loop stalling. turn it on and what caps you is the prefill budget, not the ridge batch.
the consumer card is the same equation with a lot less bandwidth. at long context the kv cache fills the card before bandwidth becomes the wall, so the ridge batch never gets a chance to matter.
Right on both, and the second one has a size to it.
Prefill: at 70B FP8 a 4k prompt is 786 ms against a 27.5 ms decode step at batch 32, so it costs 914 generated token slots. Chunk it at the 119-token break-even and it becomes 34 injections of 23 ms. The stall goes away, the 786 ms does not. At one prefill per second you are still handing 79% of the node's decode time to prefill, whole or in pieces. Turn chunking on and the prefill budget is the line, exactly as you say.
Consumer card, the number: 8B FP16 on 24 GB leaves about 8 GiB of KV at 128 KiB/token, so the card fills at 66k tokens. The KV read only equals the weight read at 122k, at batch 1. The card runs out at 54% of the way to the point where bandwidth would start to matter, which is the ridge batch never getting its turn. At b=1 the roof is 63 tok/s on a 4090 (16 GB of weights over 1008 GB/s), 58 on a 3090.
What card and model are you measuring on? I want a real consumer row next to the H100 ones.
the chunked prefill half matches what i've seen, and the prefill budget becoming the line is the part that's easy to miss. you turn chunking on, the stall goes away, and you're still handing most of the step to prompts.
on the card, the lever i'd pull before concurrency is kv precision. 8 bit roughly doubles your 66k for a quality cost i can't see in chat, and it's a flag in llama.cpp (--cache-type-k q8_0). after that the context cap per request decides, not batch, which is awkward to explain to someone who bought the card for batch.
Fair, and worth splitting in two. 8-bit KV does what you say at the cap: 66k becomes roughly 124k, not a clean double, since q8_0 quantizes in 32-value blocks with a scale per block, so about 1.88x.
But it moves the other end too. The crossover is where KV bytes per token overtake weight bytes at the ridge batch, n* = weight_bytes / (295.2 x KV_bytes_per_token), so halving KV per token doubles n*. 122k becomes about 229k. Fill and crossover both move by the same factor, so their ratio does not: the card still runs out of room at roughly 54% of the context where bandwidth would take over. FP8 KV buys context. It does not buy the regime.
Which is why I would still pull concurrency first. On a card the two are the same lever in different clothes, but concurrency you can meter and turn live, and the q8_0 accuracy cost is the part nobody in this thread has put a number on. Do you have a perplexity delta you trust, q8_0 versus f16 KV? That is the one I cannot get, and I would rather have yours than guess.
Correction to my own reply above, because I put a formula next to the wrong number.
There are two different context lengths in this and I ran them together. The first is where KV bytes catch the weight bytes at batch 1: weights/KV, which is 122k tokens for 8B FP16. The second is where the ridge batch goes to infinity: n* = weights/(295.2 x KV), which is 415 tokens on an H100. I attached n*'s formula to the 122k figure. The conclusion survives, because both scale as 1/KV, so q8_0 moves them together and the fill stays at 54% of the crossover. The sentence was still wrong while the ratio was right.
Your card, with the right number in it. ridge_AI is FLOPS/BW: 165 TFLOPS over 1008 GB/s is 164 FLOP/byte against an H100's 295. n* = weights/(164 x KV), so 8B at FP16 gives 748 tokens of context and 8B at Q4_K_M (4.9 GB) gives 228. Past 228 tokens of context, no batch size makes a 4090 compute-bound on an 8B. A system prompt clears that before your first message.
Which is the part I had wrong and now think is the whole story: KV precision cannot change the regime, only the room. The ratio of where the card fills to where the ridge dies is (free memory after weights / weight bytes) x ridge_AI, and KV bytes cancel out of it completely. q8_0 takes your 66k to about 124k and moves n* by the same 1.88x. Both endpoints move, the gap between them doesn't.
One thing you might want: quantizing the weights moves the two endpoints the other way, and apart. n* shrinks with weight bytes while the room left for KV grows, so the 8B at Q4 on a 4090 has a fill/ridge ratio of ~645x against 83x at FP16. Less precision, more room, and even less chance of ever being compute-bound. I also fixed a doubled byte factor in the script at the end of the piece, which overcounted KV by 2x for anyone who ran it.