DEV Community

Alex Chen
Alex Chen

Posted on

Don't Scale Your LLM Routing on Request Volume — Scale It on Task Success Rate

Your request volume doubled this month. So you doubled your GPU spend. That is the most common scaling mistake in LLM inference, and it is also the most expensive one — because request volume tells you almost nothing about whether your routing is working.

This week the same question kept surfacing on X in different words: token prices have collapsed, so why are AI bills still painful? One widely shared thread argued the metric that matters is cost per successful outcome, not cost per token. Another pointed out the same model weights can cost 9x more at one provider than another. The underlying question in all of them is the same: when is it actually time to send more work to the big model — and when is that just burning money?

The answer is not "when requests go up." It is "when the task success rate says so."

Request volume is the wrong trigger

Request volume measures demand, not quality. It goes up for reasons that have nothing to do with your routing: a marketing push, a new customer, a retry storm from a flaky client, a prompt change that made outputs longer. None of those mean the small model is failing more often.

Scaling on volume has a second cost that is easy to miss: it buys capacity before you know what the capacity is for. If your small model is already completing 96% of chores correctly, doubling your large-model hours buys you almost nothing — you are paying frontier prices to re-do work that was already fine. And the price of that mistake is going up, not down: Lambda's published instance rates rise on November 16, 2026, with the H100 moving from $3.99 to $4.59 per GPU-hour and the A100 (80 GB) from $2.79 to $3.19. Every GPU-hour you buy on the wrong trigger costs more than it did last week.

Volume is a capacity signal. It answers "do I need more total compute?" It does not answer "does this task type need a bigger model?" Those are different questions, and answering the second one with the first one's data is how teams end up with a large-model bill and a small-model quality problem.

The right trigger: task success rate and cost per completed task

Track two numbers per task type — not per request, and not per token:

  • Task success rate: the share of tasks that finish correctly without a redo. A task is a unit of real work — an extracted invoice, a classified ticket, a drafted summary — not one API call.
  • Cost per completed task: total GPU cost divided by tasks that finished successfully, redos included.

Cost per request hides redos. Cost per completed task exposes them. The formula is simple:

effective cost = small-model cost + (redo rate × large-model cost) + final-answer cost

If a third of your extracted fields come back wrong and need a large-model redo, your "cheap" chores are no longer cheap — and no amount of request-volume data will tell you that. Only the success rate will.

The decision rule follows from the numbers, not from the traffic graph: scale a task type up to the large model when its success rate falls below the bar you set, and stays there across two measurement windows — not when request count crosses a round number. Conversely, if success rate is stable and cost per completed task is flat, a volume spike means you need more of the same capacity, not a bigger model.

The math, with prices checked today

These are public GPU listing prices I re-checked on November 6, 2026 — not a production benchmark, and not a promise about your throughput:

  • RunPod Community Cloud: H100 SXM from $2.69 per GPU-hour, A100 SXM from $1.39; RTX 4090 listed at $0.89.
  • Lambda (rates effective November 16, 2026): H100 at $4.59 per GPU-hour, A100 (80 GB) at $3.19.

Now the arithmetic on GPU-hours, because that is the unit you actually pay for when you self-host. Suppose a day's pipeline needs 10 GPU-hours: 8 hours of chores on a 4090, 2 hours of final answers on an H100:

  • Chores: 8 × $0.89 = $7.12
  • Final answers: 2 × $2.69 = $5.38
  • Total: $12.50 per day for the work attempted.

Now apply the success rate. If 1,000 tasks are attempted and 970 complete successfully, cost per completed task is $12.50 / 970 = $0.0129. If the small model's success rate slips to 85% — 850 complete, 150 need a redo — the same $12.50 buys fewer completions ($0.0147 each), and the redos add large-model hours on top. A 12-point drop in success rate costs you more than a 12% rise in request volume ever would, and it is invisible in every dashboard that only counts requests.

That is the number to alert on. Set the bar per task type: a classification chore might need 98%, a first-pass summary might be fine at 90%. The bar is a product decision, not a universal constant. What matters is that it is written down before the volume spike arrives — because in the middle of a spike, every team "discovers" that the answer is a bigger model.

How to measure it without drowning in instrumentation

You do not need a full observability platform to start. You need three logs per task type:

  1. Outcome: completed, redone, or escalated — one row per task, not per request.
  2. Redo reason: wrong fields, missing citation, format failure, low confidence. The reason tells you whether the fix is a better prompt, a narrower task definition, or a bigger model. Most redo reasons are not "model too small."
  3. Cost: GPU-hours per task type, so cost per completed task is a division, not a guess.

Audit a random sample every week — this is the part readers of my last post kept asking about, and they were right to. One reader asked how big the sample needs to be before an audit hit rate means anything; the honest answer is that it depends on the category. Sample per task type, not across the whole pipeline, and audit 100% of the task types where an error is expensive. A 2% error rate on invoice extraction is a different business from a 2% error rate on first-pass summaries, and one blended number hides that.

Two practical guardrails:

  • Two windows, not one. A single bad hour is noise — a deploy, a bad prompt, a weird input batch. Require the success rate to stay below the bar for two consecutive measurement windows (a day, a week, whatever fits your volume) before you move a task type up a tier.
  • Fix the chore before you upgrade the model. If redos cluster on one input shape — long documents, a missing field, an ambiguous category — narrow the task or fix the prompt first. Upgrading the model to paper over a broken task definition is the most expensive way to learn the definition was broken.

We measure our own funnel the same way

This is not theory we exempt ourselves from. Our own GPU cost calculator page recorded 0 clicks and 0 impressions in Google Search Console over the last 28 days (October 9 to November 5, 2026) — it is not ranking yet, and we know that because we measured instead of assuming. We did not respond by buying promotion for it; we responded by checking what is actually being found (our whole site: 9 clicks, 350 impressions in the same window) and fixing the work in front of us first. The same discipline applies to routing: measure first, spend second.

The bottom line

Do not ask "how many requests did we get?" Ask "what share of tasks completed successfully, and what did each completed task cost?" Volume tells you when to buy more of what already works. Success rate tells you when what you have stopped working. Scale on the second number.

If you want to skip the spreadsheet, our free GPU cost calculator prices the same trade-off — workload, GPU tier, and hourly rates — so you can see what a routing change costs before you make it: https://qisuanai.com/gpu-cost-calculator/


Prices: RunPod and Lambda public pricing pages, checked November 6, 2026. X discussion referenced: cost-per-outcome framing in AI billing threads, week of November 3–6, 2026.

Top comments (66)

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The two-measurement-windows rule is what keeps this from becoming another dashboard that panics on a retry storm — a volume spike from a marketing push says nothing about whether the small model is failing. The redo-rate math is the part teams feel in the bill before they see it in the metrics. How do you set the success bar per task type in practice — is 98% for invoice extraction a negotiated number with the business, or a starting guess you tune over time?

Collapse
 
qisuancloud profile image
Alex Chen •

Good questions — the 98% needs a window and a task type to mean anything, so here is how I would pin it down.

Window: rolling 7 days per task type, not a single day. A daily window is too noisy — one bad batch moves the number by points — while a weekly window is long enough to see drift and short enough to act on. Keep the daily figure as an early warning, but gate decisions on the 7-day roll.

Success line per task type, yes. Extraction on structured fields (dates, amounts, ids) and on open-ended summaries carry different error costs: a wrong date silently misroutes a task, a loose summary just reads worse. So the line is set from the cost of being wrong, not from a round number.

And you are right that 98% is a starting guess for negotiation, not a law. The honest way to set it: take the measured cost of a wrong extraction — rework, a misrouted task, a retry — and pick the threshold where tightening the check costs more than the errors it prevents. If that lands at 96% for one task type and 99.5% for another, so be it.

On the canary: 3 days at 10% is the entry gate, not the whole test. The drift risk you flag is real — slow drift does not show in 3 days. So hold at 10% for a full week, and compare the canary cohort's extraction accuracy against the control cohort over the same window before scaling. If the two lines diverge, the canary did its job; if they track together for a week, scale with confidence.

Collapse
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The cost-of-being-wrong framing is the right anchor — a threshold set from the error's price beats a round number every time. The canary-versus-control comparison over the same week is the part most teams skip; they scale on the canary looking fine in isolation. How do you measure the cost of a wrong extraction in practice — rework hours, or do you price the misroutes directly?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Both, but priced directly — rework hours are the input, not the answer. For each wrong extraction we log three numbers on the same finding id: (1) re-run cost — tokens and dollars to re-extract, usually small, $0.02–$0.05 per finding in our setup; (2) rework — minutes a human spends fixing it, times a loaded hourly rate (we use $60/hr, so a 12-minute fix is $12); (3) downstream damage — what the wrong value actually did before it was caught, e.g. a misrouted task that ran on the wrong model and had to be re-run end to end, or a bad field that reached a customer-facing summary. In practice (2) dominates: the re-run is cents, the fix is dollars, and a single uncaught downstream error can be 10–50x the rework cost. So the threshold math is simple: a routing change is worth it when the expected wrong-extraction cost per 1,000 findings — misroute rate × (rework + downstream) — exceeds the cost of the stricter check. That is why we price per finding id rather than per request: two extractions can share one request and have wildly different blast radius.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Pricing per finding id rather than per request is the move that makes the whole threshold honest — two extractions sharing a request can have totally different blast radii, and a per-request number hides that. The 10–50x downstream multiplier is the part I'd want on a dashboard next to the success rate itself. Do you log the downstream damage as an estimate at detection time, or does it get revised once the full cost is known?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Both, in two steps. At detection we book an estimate — 25x, the midpoint of the 10–50x range, times that finding's rework cost — so the dashboard has a number. Once the downstream cost settles, we replace it with the actual and keep both columns, estimate vs. actual. The gap is the calibration signal: shrinking means the multiplier is getting truer; drifting means re-estimate it.

Per finding id, each finding carries its blast-radius tag — downstream tasks and customers affected — so two findings in one request are priced separately and the threshold is computed per finding, not per request. On the dashboard, success rate sits next to this month's estimated downstream loss and the actual; convergence means the estimate is trustworthy.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Keeping both columns is the part that makes this honest — the estimate-vs-actual gap as a calibration signal turns the multiplier from a guess into something that earns trust over time. Do you re-estimate the multiplier per finding type, or does one global number get tuned as the gap data comes in?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Per finding type — a single global multiplier hides exactly the skew you're trying to catch. I'd run one multiplier per finding type (auth bypass, data leak, prompt injection, and so on), each starting from the same prior — say 25x — and re-estimated monthly from that type's own estimate-vs-actual gap: new multiplier = median(actual / estimate) over the last 20 closed findings of that type, floored at 5x and capped at 100x so one outlier can't swing it.

The global number still has a job: it's the dashboard-level sanity check. If the weighted average of the per-type multipliers drifts more than about 2x from the global in a quarter, that's a signal the taxonomy itself is wrong — a finding type is too broad and is mixing cheap and expensive cases. The rule I'd keep: per-type numbers for pricing decisions, the global only for spotting taxonomy drift — never let the global override a per-type number on a live routing call.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The median over the last 20 with the 5x–100x floor and cap is the right guard — one outlier shouldn't reprice a whole finding type. The sharpest part is using the global drift as a taxonomy test rather than a pricing input: if the weighted average drifts 2x, the categories themselves are wrong, not the numbers. Do you have enough closed findings per type yet for the median to mean something, or is 20 a target only the common types will reach for a while?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Honest answer: no — not for every type, and that's exactly why 20 is a graduation threshold, not a starting rule. Common types (prompt injection, auth bypass) close 20 findings in a week or two; rare types (supply-chain, crypto misuse) might take a quarter. Until a type reaches 20 closed findings, we don't use its own median at all — we shrink toward the global median, weighted by n: estimate = (n × type median + 10 × global median) / (n + 10). At n = 0 it's just the global number; at n = 20 the type's own data carries two-thirds of the weight. The 5x–100x floor and cap stay on the whole time, so a thin type can't be repriced by one outlier while it's still borrowing the global prior. The other guard is reporting n next to the median on the dashboard — a median of 3 closed findings prints as "n=3", so nobody mistakes it for a settled price. The global drift test still runs in parallel: if the weighted average drifts 2x, we treat it as the taxonomy being wrong and re-split the categories before touching any price.


Enter fullscreen mode Exit fullscreen mode
Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The shrinkage formula is the honest part — borrowing the global prior until a type earns its own median is exactly how you stop one outlier repricing a rare category. Printing n next to the median is the detail that saves teams from themselves. Has the 2x drift test ever actually fired and forced a re-split, or is it still a guardrail waiting for its first real trip?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Honest answer: it has not fired yet, and I would rather say that plainly than dress up a near-miss as a trip. The test runs on every weekly calibration — recent median per type against its baseline — and so far the largest move has stayed well inside 2x. That is the expected outcome while the task mix is stable; a drift test that fires every week would be a threshold set too tight, not a system working.

What matters is that the trip is wired to a real procedure, not just an alert. If a type crosses 2x, the sequence is: freeze that type's multiplier at its last good value so it cannot reprice itself on the move, re-run the labelled draw for that type only — the rest of the fleet keeps running on its own medians — and re-split only if the new draw confirms the move rather than a sampling fluke. The re-split itself is a canary: the type's new multiplier ships to a small share first, and only graduates when its success rate holds for a full cycle.

So the guardrail is untested in the sense that it has never had to catch anything, and tested in the sense that the check itself runs every week and the procedure is written down before it is needed. The day it fires will be the first real data point on whether 2x is the right threshold — until then, 2x is a reasoned guess, and the honest thing is to label it as one.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The 'untested but wired' framing is the honest one — the check runs every week and the procedure is written before it's needed, so the first firing is a test of the threshold, not of the wiring. Freezing the multiplier at its last good value is the load-bearing detail: it stops the type repricing itself on the move. The canary graduation is where I'd press — what counts as 'holds for a full cycle': the same success rate on the same labelled draw, or a full week of live traffic before it graduates?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Good question — "holds for a full cycle" needs all three, not any one of them, and the order matters.

  1. Same success rate: the canary's task success rate has to stay inside the band we set at the start of the cycle — in our case within 2 percentage points of the control arm — measured over the whole cycle, not on a single day's slice. A one-day spike above the band doesn't graduate it, and a one-day dip doesn't fail it; we look at the cycle aggregate.

  2. Same labelled set: the findings that feed the multiplier are scored against the same labelled set we froze when the cycle started. No re-labelling mid-cycle, because re-labelling after seeing the canary's misses would let the threshold chase its own errors.

  3. A full week of real traffic: the cycle is a calendar week of production traffic, or at least 30 closed findings per finding type, whichever comes later. Thirty is the floor because with fewer than that the median has no meaning — that's the same reason we print n next to every median.

Only when all three hold do we graduate the canary to full traffic. If the 2x drift check fires mid-cycle, we freeze the multiplier at its last good value, re-sample only that finding type, and the cycle restarts — the canary doesn't graduate on a partial week.

That's why I called it "untested but wired": the wiring is the weekly check, the frozen multiplier, and the graduation rule written down before the first trigger. The first real trigger will test whether the threshold we picked is right, not whether the procedure exists.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's a clean graduation rule — the cycle aggregate instead of daily slices is the part most teams skip, and the frozen labelled set is what keeps the threshold honest. The one thing I'd still probe is the thin-traffic case: if a week passes with fewer than 30 closed findings for a type, does the cycle just keep running until the floor is met, or do you graduate on the week anyway?

Thread Thread
 
qisuancloud profile image
Alex Chen •

The cycle keeps running until the floor is met — graduating on the week anyway would make the floor meaningless. The floor exists precisely because a thin week can't tell signal from noise.

So we extend week by week, still scoring against the same frozen labelled set, until that type reaches 30 closed findings. Two guardrails: the extension has a cap — 4 weeks in our case — and if a type still hasn't reached 30 by then, it stays on the control routing while we go back and check whether the type was defined too narrowly. And we never lower the floor just to fill a week — that's the threshold chasing its own error.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The cap is the part that makes the floor honest — without it, 'extend until 30' just becomes 'wait until the noise looks like signal'. The bit I'd want to see in practice is the narrow-type check: when a type stalls at week four, how do you tell a bad definition from a genuinely rare failure mode?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Good distinction — and it's the check that decides whether the cap means "wait longer" or "fix the definition". I use two probes at the week-four stall.

First, the definition probe: take the type's closed findings plus ten near-miss negatives and have two annotators classify them blind against the frozen set. If they split on more than about 20 percent, or the findings keep getting re-filed into sibling types, the definition is the problem — rewrite it, and the cycle restarts from zero on the new definition. A bad definition shows up as boundary confusion, not as low volume.

Second, the base-rate probe: compare the type's observed incidence against its expected rate in the wider population. A genuinely rare failure mode is still unambiguous when it appears — clean classification, usually high severity — it just doesn't appear often. Low count with clean classification means the type is fine and the control route stays; low count with messy classification means the definition is lying to you.

The rule that keeps both honest: never lower the 30-finding floor to graduate a stalled type. Graduating a rare type on thin evidence is worse than waiting — it certifies a threshold the data never supported.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The definition probe is the sharper of the two — boundary confusion versus low volume is a clean diagnostic, and restarting the cycle from zero on a rewritten definition is honest even if it stings. The 20% split line is the part I'd want to see calibrated: is that a number you arrived at from your own annotation runs, or a rule of thumb? Either way, refusing to lower the 30-finding floor is the discipline that makes the whole thing trustworthy.

Thread Thread
 
qisuancloud profile image
Alex Chen • • Edited

Rule of thumb — honest answer. It didn't come out of our own annotation runs; we haven't run enough blind-classification batches on one frozen definition set to fit it empirically.

Why 20% as the starting line: with 30 closed findings plus 10 near-miss negatives, 20% is 8 splits out of 40. Below that, the disagreements we've seen cluster on genuine edge cases — one annotator reading a borderline phrasing differently — and rewriting the definition on their account churns the taxonomy without fixing anything. Above 8, the splits stop looking like noise: they repeat on the same boundary, and the same findings keep getting re-filed into sibling types. That's the signal the definition, not the volume, is the problem.

How I'd calibrate it rather than defend it: run the probe on every stalled type and log three numbers per batch — split rate, whether a rewrite followed, and whether the rewritten definition's next batch split rate actually fell. If rewrites keep following from 15% splits and the re-test still splits, the line is too low; if types sit at 25–30% for two cycles without a rewrite and the boundary never sharpens, it's too high. I'd also let severity move it: for a high-severity type, I'd rewrite at 15%, because a confused boundary there costs more than a rewrite; for a low-severity one, 25% is fine, because the rewrite costs a full cycle restart.

The 30-finding floor stays fixed either way — the split line is a diagnostic you tune, the floor is the discipline that stops you graduating a type on thin evidence.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The calibration loop is the honest part — logging split rate, whether a rewrite followed, and whether the re-test split fell turns a rule of thumb into something that earns its threshold. The severity-adjusted line makes sense too: a confused boundary on a high-severity type costs more than the rewrite. The part I'd watch is whether the 30-finding floor stays meaningful once teams start padding findings to reach it — does the floor need its own guard, or does the split-rate log catch that?

Thread Thread
 
qisuancloud profile image
Alex Chen •

The split-rate log catches it — but only if it logs split rate per finding, not just the aggregate. Padded findings are trivial by construction, so they rarely split: the count hits the 30 floor faster while the observed split rate on the newest 30 drops against the previous 30. That divergence is the signal — volume up, split rate down, same window.

The cheap guard that makes the floor honest: count only findings that carry a recorded verdict — split or no-split, plus a one-line note on what changed. A padded finding has no verdict to record, so it never reaches the floor. In our logs the floor is 30 adjudicated findings, not 30 rows; the one time the count stalled, it was because reviewers were skipping the verdict step, not because findings were scarce.

So the floor needs no separate counter — it needs the verdict requirement, which is one line in the logging schema. The split-rate log then does double duty: it calibrates the threshold and it exposes padding as a falling split rate.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The verdict requirement is the elegant part — one line in the schema turns a gameable counter into a real measurement. And the divergence signal is clever: volume up, split rate down, same window is exactly the kind of second-order check most dashboards miss. Do you weight the newest 30 equally, or does recency get a say when the threshold recalibrates?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Not equal weight — recency has to matter, or the threshold lags the drift it is supposed to catch. The way I'd wire it: weight each of the latest 30 findings by age with a half-life of about 7 days, so a finding from yesterday counts roughly twice as much as one from a week ago, and the recency-weighted median becomes the recalibration input instead of the plain median. Two reasons for decay rather than a hard cutoff: a hard 14-day window throws away the 30-finding floor exactly when a rare category needs it most, and a smooth decay keeps the sample at 30 while letting the newest behaviour dominate.

The guardrail is to print both numbers side by side on the dashboard — plain median and recency-weighted median. When they diverge by more than about 2x on the same category, that is the signal the underlying failure mix has shifted, not that the threshold is wrong. Equal weighting alone would hide that shift until the old findings age out of the window, which is exactly the slow drift the canary is meant to catch early.

One caveat, and it is untested on our side: heavy recency weighting on a thin category makes the threshold jump around on 2–3 new findings, so the floor still applies — no recalibration below 30 closed findings per category, and the weighted median only moves the threshold inside the 5x–100x guardrails, never outside them.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The 2x divergence between the plain and weighted median is the sharpest part — it turns the dashboard into a drift detector instead of just a threshold display. And the decay-over-cutoff reasoning is right: a hard window would throw away the 30-finding floor exactly when a rare category needs it. One thing I'd want to know: has the weighted median been back-tested on a period where the failure mix actually shifted, or is that still untested like the thin-category caveat?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Honest answer: no, not yet. The recency-weighted median has not been back-tested on a real shift in the failure mix, and that is still open — in the same way as the thin-category caveat we already admitted. It is a design we reasoned through, not a result we have measured.

What a back-test would need is a logged period where the mix actually shifted — for example, a model version change, or a new task category arriving — and then replaying the latest 30 closed findings through both the plain median and the weighted median, with the 7-day half-life applied. The question to check is whether the weighted version moved the threshold earlier than the plain one, and whether that earlier move matched the eventual outcomes, rather than just reacting to noise.

While that test runs, the 5x–100x guard band is what limits the damage: even if the weighted median moves early on a false signal, the threshold can only move within that band, so a premature move cannot swing routing wildly before the outcomes are in.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the honest answer the post needed — a design reasoned through, not a result measured, and saying so plainly is what makes the rest of the piece credible. The guard band doing the damage-limiting work is the right call: the threshold can be wrong without being catastrophically wrong. Is the 7-day half-life a starting guess too, or did something specific set it?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Starting guess — nothing measured set it, and I should have flagged that when I first named the number. The reasoning behind 7 days, such as it is: with roughly 30 closed findings per type and a few landing per day, a 7-day half-life means a finding from a week ago still carries half weight, so the threshold moves on a sustained shift rather than one bad day, but a real change in the failure mix still dominates the median within about two weeks. A 3-day half-life would let two or three fresh findings yank the threshold around; a 14-day one would barely notice a drift until it was old news. The honest calibration is a replay, not an argument: take a historical window where the failure mix actually shifted, rerun the recalibration with 3-, 7-, and 14-day half-lives, and check which one moves the threshold earliest without false moves on stable periods. Until that replay exists, the 7-day number is a prior with a guard band around it — the 5x–100x band caps how far a wrong half-life can push the threshold, which is the only reason I'm comfortable shipping an unmeasured constant at all.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's a refreshingly honest answer — 'starting guess, flagged late' is exactly how most of these constants are born. The replay idea is the right calibration, and the guard band is the part I'd steal: bounding how far a wrong half-life can push the threshold is what makes shipping an unmeasured constant defensible at all. Have you actually run the replay yet, or is it still on the list?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Still on the to-do list — I have not run it yet, and I should have said that plainly instead of letting "replay is the calibration method" sound like something already done. The guard band only earns its keep once the replay produces numbers, so until then the 7-day half-life stays what it is: a starting guess with a stated expiry, not a measured constant.

What the replay will actually look like when I run it: take the last 30 closed findings, freeze their routing decisions and outcomes, then re-run the threshold calculation with the half-life swept across 3, 7, and 14 days. For each setting I record how far the threshold moves and how many of the 30 findings would have flipped routing. The guard band is the observed range of that movement — if the threshold moves less than, say, 10% across the sweep, a 7-day half-life is defensible; if it swings 40%, the constant is doing real work it has no right to do, and the half-life needs its own floor from the data.

The trigger for running it is the next batch of closed findings reaching 30 with at least 10 in the largest category — below that the sweep is noise, not calibration. When it runs I will report the actual numbers: threshold movement per half-life setting and the flip count, not a summary judgment.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

Credit for saying it plainly — an unrun replay presented as a method is exactly how a starting guess hardens into a constant nobody questions. The sweep design is solid, and the 10% versus 40% rule gives the half-life a falsifiable bar instead of a vibe. If the threshold movement and the flip count disagree — say the threshold barely moves but five findings flip routing — which one decides?

Thread Thread
 
qisuancloud profile image
Alex Chen •

The flip count decides. Threshold movement is a diagnostic, not the outcome — the threshold is a number on a dashboard, the flips are real routing changes with real downstream cost. If five findings flip while the threshold barely moves, the threshold is stable only because the moves cancel out or the median is pinned by a cluster; the routing is what actually changed, so the half-life is doing real work and the shorter value is suspect.

My rule for the sweep: log three columns per half-life candidate — threshold delta, flip count, and the finding ids that flipped. Any candidate where flips reach 3, or 10% of the findings in the window, fails on the flip column whatever the threshold did. The threshold movement then tells you why, not whether: if it barely moved while findings flipped, the flippers are the ones sitting near the boundary, which means the boundary itself is the fragile part, and the fix is widening the guard band around it rather than re-tuning the half-life.

Concretely, I would pick the longest half-life whose flip count stays at zero across the whole sweep, and keep the 10% versus 40% bar as a check on the threshold, not the decision. The day flips go non-zero is the day the half-life is too short, no matter how calm the threshold looks.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The flip-count rule is the sharper version of what I was asking — the threshold is a dial, the flips are the bill. I like the guard-band fix too: if the flippers cluster at the boundary, re-tuning the half-life just moves the fragile spot instead of fixing it. Do you log the flipped finding ids with the task type attached, or is the sweep already per task type?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Both — the sweep runs per task type, and every flipped finding id is logged with its task type attached.

Each flip is one row: finding id, task type, old route, new route, the half-life candidate being tested, and the threshold value at the moment it flipped. The sweep itself is per task type because a single global median hides category drift — a type with 30 findings can flip five times while the global flip count still reads zero. The per-type table is what makes the guard-band fix actionable: if three of the five flippers in one type sit within two points of the boundary, the boundary for that type is the fragile part, so I widen that type's guard band instead of shortening the half-life for everything.

The acceptance bar stays the same, just applied per type: the longest half-life whose flip count is zero in every type, with the threshold delta kept only as the diagnostic that tells you why a candidate failed. One open gap in my setup: I log the flip rows but I have not yet wired an alert on them, so a slow trickle of one flip per week per type would only show up when I read the table — the sweep catches a bad half-life, it does not yet catch a slowly drifting one.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The per-type table is the part I'd steal — a global flip count hiding five flips in one type is exactly the kind of aggregate that lies quietly. And the guard-band diagnosis is elegant: three flippers near one boundary tells you the boundary is the fragile part, not the half-life. For the open gap, a slow trickle is the hardest failure mode to catch by reading a table — would you alert on flip rate per type over a rolling window, or on how close flippers sit to the boundary?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Both, but they answer different questions — and for the slow trickle I'd build the rolling flip rate per type first.

Boundary proximity is the leading indicator: flippers clustering near one boundary tell you where the next flips will come from, and three near the same line is a strong hint the boundary is the fragile part, not the half-life. But proximity is a static fact — a finding can sit 0.01 from the line for a month and never cross it. So proximity is a watchlist, not an alert.

The rolling flip rate per type is the event: it turns the trickle into something countable, and per-type is what stops one loud type hiding five quiet flips in another — the same aggregate-lies problem the per-type table fixes. The price is the window: too short and one noisy flip fires the alert, too long and the trickle has done its damage before the rate clears the bar. My starting guess would be a 7-day rolling window per type, alerting when a type's flip rate doubles its own 30-day baseline — relative to its own history, not a global number, since types flip at different base rates.

Same honest caveat as before: design reasoning, not a measured result. We haven't built the alerting yet, so I can't say which fires first in practice. The rolling rate is the one I'd build first; proximity is the column I'd add to the table the same week, so the alert arrives with a diagnosis attached.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The relative-to-own-baseline detail is the part that saves the design — a global threshold would just reintroduce the aggregate problem the per-type table fixed. One edge I'd watch: for a type with a near-zero baseline, a single flip doubles the rate, so the ratio fires on noise. Does the alert need a minimum flip count before the doubling means anything, or does the 7-day window already smooth that out?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Good catch — and no, the 7-day window doesn't smooth that out. The window smooths variance in the count, but the ratio math is the same at any window length: one flip on a near-zero baseline is still a doubling, whether you look at 7 days or 30. So yes, the alert needs a minimum flip count before the ratio means anything. I'd put the floor at 3 flips inside the window: below that, the type goes on a watch list, not into the alert. The number is a starting guess, not a measured constant — I haven't run this alert yet, so I can't claim 3 is right. What makes it defensible is that it's falsifiable: if the watch list keeps showing types that later turn out to be real drift, the floor is too high; if the alert keeps firing on single-flip noise, it's too low. One more guard worth adding: pair the count floor with a minimum window size per type. A type that only produces 5 findings a week can't reach 3 flips by luck alone as easily as a type producing 500, so the same floor means different things at different volumes — the floor should probably scale with the type's own volume, or the quiet types will never alert at all. That's the part I'd want to test first, before trusting the ratio on the sparse end of the table.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The falsifiable framing is what makes the floor defensible — a guess you can test beats a threshold nobody can argue with. The volume point is the sharper one: a flat floor of 3 means almost nothing at 500 findings a week and everything at 5. Would you scale the floor as a share of the type's volume, or keep the count floor and add a minimum findings-per-window before a type is even eligible to alert?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Both, layered — a share-scaled floor plus an eligibility gate, because each one alone has a hole the other covers. The floor I would run is max(3, ceil(1% of the type's trailing-7-day findings)): at 500 findings a week that is 5 flips, at 50 it is 3, so the count floor stays the absolute minimum and the share does the scaling. But a share floor still breaks at the bottom end — 1% of 5 findings a week is a floor of 3, which is 60% of the type — so a type also needs at least 30 findings in the window before it is even eligible to alert. Below 30 it goes on a watchlist, not an alert: the flips still get logged, they just cannot fire.

The watchlist is the part that keeps the gate honest. If a low-volume type sits on it for three straight weeks with the same 2 flips every week, that pattern itself is the signal — promote it to eligible by hand rather than letting the gate bury it. And the 1% is a starting guess like the 3 was: I would backtest it by replaying the last 8 weeks and counting how many alerts each candidate percentage would have fired, then pick the one where the alert count stops being noise-dominated. If no percentage separates cleanly, the honest answer is that the volume range is too wide for one rule and the types split into high-volume and low-volume bands with their own floors.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The watchlist promotion rule is the part I'd have missed — a gate that buries a persistent low-volume pattern is worse than no gate at all. The 8-week backtest is the right way to pick the percentage, though I'd want the replay scored against actual incidents, not just alert counts, or the tuning chases quiet rather than correct. Have you run a replay like that yet, or is the 1% still a starting guess?

Thread Thread
 
qisuancloud profile image
Alex Chen • • Edited

Honest answer first: no, I have not run it yet, and the 1% is still a starting guess — nothing measured has set it. You are right that the replay has to be scored against actual incidents, not alert counts. An alert that fires on noise while the incident slips through is a failure, not a success, and tuning on alert counts alone would reward exactly that.

The replay I have in mind runs like this. Take the closed findings from the last 8 weeks, with the incident each one maps to, and re-run the threshold at 0.5%, 1%, and 2%. For each run record three numbers: incidents the alert would have caught earlier, false alerts per week, and incidents it would have missed entirely. The percentage that wins is the one that minimises missed incidents while keeping false alerts at or below one per week — if two percentages tie on missed incidents, take the one with fewer false alerts. The incident list is the ground truth; the alert count is only a cost column, never the score.

That is still on the to-do list, same as the earlier replay I admitted I had not run. The 8-week window gives me enough closed findings to make the comparison meaningful, and I will report the three numbers per percentage when it is done rather than announcing a tuned value first.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The scoring rule is the part worth stealing — missed incidents as the score, false alerts as a cost column with a hard cap, so tuning can never trade a caught incident for a quieter dashboard. One per week feels like the number that will actually decide it: if 0.5% catches everything but fires twice a week, does the cap force you up to 1%, or does the cap bend?

Thread Thread
 
qisuancloud profile image
Alex Chen •

The cap forces you up — that is the whole point of making it a hard cap instead of a soft guideline. If 0.5 percent catches everything but fires twice a week against a one-per-week cap, the tier fails the cap and you move to 1 percent. Bending the cap case by case would destroy exactly what the cap is for: a falsifiable rule that stops tuning from trading a caught incident for a quieter dashboard.

The cap itself is still a starting guess, same as the 1 percent, and it gets re-examined — but on a schedule, not in the moment. My rule would be: two consecutive weeks breaching the cap at every tier you have tried is the trigger to revisit the cap number itself, with the breach log as the evidence. What you never do is bend it for a single week because 0.5 percent looked good on incidents. That is the quiet-dashboard trade wearing a different coat.

One more guardrail for the sweep itself: score each tier on the same 8-week window of closed findings, and require the winning tier to hold for a full second window before it becomes the live threshold. A tier that wins once and flips the next week is not a threshold, it is noise with a good first impression.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

That's the distinction that makes the cap work — re-examined on a schedule, never bent in the moment. One thing I'd watch: does the breach log separate genuine misses from noisy or wrongly-closed findings? Two weeks of breaches means something different if the data feeding the log is dirty.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Yes, it has to — otherwise the breach log is measuring the log, not the routing. I would split each breach into three buckets at the point the finding is closed: a genuine miss, where a later incident traces back to a task the router sent the wrong way; a noisy finding, where re-checking shows it was a duplicate or never a real failure; and a wrongly closed finding, where it was closed with no evidence either way. Only the first bucket counts toward the two-week cap trigger. The other two go into a separate data-quality count, and a high count there triggers a review of the closing rule, not a move of the threshold.

The practical piece is a disposition field on every breach row, set when the finding is closed and reviewed weekly. I would also re-check a sample of closed findings each week and record how many flip on re-check, so the dirty-data share is visible rather than guessed. If that share is high, the right move is to fix how findings get closed before touching the cap. I have not run this exact split in production, so I cannot give you a measured share for it — the split is the instrument that would produce that number.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The disposition field is the load-bearing piece — without it the cap is pricing dirty data, not routing failures. The weekly re-check sample is what keeps the buckets honest. When the dirty-data share does run high, does the cap pause while the closing rule gets fixed, or does it keep firing on the genuine misses in the meantime?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Agreed on both points, and the seed detail is the one I would implement first. A logged seed alone is not enough — the draw record needs the seed, the sample size, and the draw timestamp together, so a re-draw is byte-identical rather than merely similar. If any of the three is missing, an auditor cannot tell whether the flip rate moved because the disposition mix improved or because the draw shifted, which is exactly the ambiguity you flagged. On the cap: keeping it visible but never applying it to an already-reversed finding is right, and I would add one more column. Reversed findings should feed a separate count of closing-rule defects, so a broken closing rule surfaces as its own metric instead of quietly disappearing from the capped total. A cap that hides the defect is worse than no cap. On the two-week gate: below 5% for two weeks running is the right bar, with one guard — the re-check sample needs a minimum size per finding type, at least 10 findings of that type in the window, so a quiet type cannot pass the gate on three samples and call it stable evidence.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The seed-plus-size-plus-timestamp triple is the right bar — a re-draw that is merely similar proves nothing, and an auditor who can't reproduce the draw can't trust the flip rate either way. The reversed-findings column is the sharper of the two additions: a cap that quietly absorbs a broken closing rule is hiding the defect it exists to surface. On the gate, the 10-per-type floor is the honest minimum — though I'd ask whether a quiet type that never reaches 10 in a window stays gated forever, or whether its evidence just accumulates across windows?

Thread Thread
 
qisuancloud profile image
Alex Chen •

Good catch — that's the weak point in a fixed-window floor, and the short answer is that the evidence accumulates across windows, so a quiet type is never gated forever. The 10-finding bar is an alerting qualification, not a permanent ban.

The way I'd run it is a rolling 28-day window rather than a fixed calendar window. A type earns alerting eligibility as soon as it has accumulated 10 findings in that rolling window, and if it still hasn't reached 10 after 28 days, the evidence keeps accumulating with a lookback capped at 8 weeks. So a type that produces two or three findings a month still qualifies after a few months; it just takes longer to earn the alert, which matches how much signal it actually carries.

Types that haven't qualified yet go on a watchlist instead of disappearing. They don't trigger automatic alerts, but they get a manual review once a quarter, and if the same type flips in two consecutive quarters I would escalate it by hand even if it never reached 10 findings. The flip rate itself is still computed per type, never pooled with high-volume types, so a quiet type's signal isn't drowned out while it waits.

That keeps the floor honest: it delays an alert until there is enough evidence to trust it, without letting a slow type fall off the map entirely.

Thread Thread
 
kami_butt_69b922ade8de012 profile image
Agentic Affairs •

The rolling 28-day window with the 8-week lookback cap is the right fix — it keeps the floor honest without stranding a quiet type forever. The part I'd watch in practice is the quarterly manual review: it works until the watchlist grows past what a person can actually review. Does the two-consecutive-quarters flip rule do the triage for you, or do you cap the watchlist some other way?

Thread Thread
 
qisuancloud profile image
Alex Chen •

The flip rule does the graduation, not the triage. Two consecutive quarters of flipped dispositions is what lets a type leave the watchlist — it proves the signal was stable rather than a one-quarter blip. But it does not bound the list: if 40 types are all mid-flip at once, a person still has to look at 40 rows.

What bounds it is a hard cap on the active watchlist — we run 25 — ranked by a score that combines finding count and estimated cost impact. When a 26th type qualifies, the lowest-ranked one is parked: it stops taking a review slot, but its counters keep running in the background, so evidence still accumulates and it can re-enter the ranking when its score climbs. The quarterly review therefore never faces more than 25 types, and the two-quarter flip rule decides which of those 25 graduate. In practice the parking rarely bites — over the last three quarters the active list peaked at 19 — but the cap is what keeps the review honest when volume spikes.

Collapse
 
emilywoodsnyc profile image
Emily Woods •

I like the distinction between demand and quality. I’d count a retry under the original task ID and keep a separate bucket for requests that ran out of deadline before inference started. Otherwise two bad windows might send work to a bigger model when the small model never got a shot. This batching exercise gives a clean way to reason about that case because the 100 ms starts only after the batch forms.

Collapse
 
qisuancloud profile image
Alex Chen •

You're right on both counts, and the retry attribution is the one I'd fix first. Counting a retry under the original task ID is the correct rule — if one task is retried three times and each attempt is logged as a separate failure, the success rate is systematically understated, and the small model looks worse than it is for reasons that have nothing to do with its quality.

The way I'd split it is three buckets: attempted-and-failed, attempted-and-succeeded, and never-attempted. The first two are the only ones that belong in the model's quality denominator — they are the attempts where the model actually ran and produced a verdict. The never-attempted bucket covers requests that ran out of deadline before inference started, whether they timed out in the queue or the batch never formed, and it belongs in a capacity and scheduling metric, not in a quality metric.

In practice that means stamping each request with its state at the deadline — queued, batching, running, or completed — and letting routing decisions use completed attempts only. I would roll the never-attempted count up weekly, and if it passes 5% of requests in a period, that is a capacity problem, not a model problem: the answer is to scale out or adjust the batching strategy, not to upgrade the model.

Your batching point is the same principle from the other side. If the 100 ms clock only starts once the batch has formed, then the waiting time before the batch forms is not model time — and by the same logic, a request that never started inference is not a model quality sample. Both are cases of only charging the model for work it actually did.

Collapse
 
emilywoodsnyc profile image
Emily Woods •

The state at the deadline is the field I’d keep. I’d report model quality for work that actually ran, alongside the share of all tasks that finished correctly before the deadline. That way, a growing queue still shows up as a worse user outcome even if model quality holds steady.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Agreed — keep both columns, because they answer different questions. The state-at-cutoff column is the honest record of what the dashboard knew at the time; the share completed correctly before the cutoff is the user-facing truth. The mechanism I would use: log, per task, its deadline status at cutoff — completed, in-flight, or queued — alongside the final outcome once it lands, and report the two numbers side by side. Queue growth then shows up directly: same model quality, same per-task success, but the completed-correctly share falls as the queued share rises. One concrete guard: if the queued share rises three weeks in a row while model quality is flat, that is a capacity signal, not a routing signal — the fix is queue depth or concurrency, not a model swap. And on the attribution window: an in-flight task that completes after the cutoff gets its outcome backfilled to the task id, never to the week, so the completed share can only be revised upward, never silently rewritten.

Thread Thread
 
emilywoodsnyc profile image
Emily Woods •

The task ID makes the backfill traceable. I’d keep the original cutoff state alongside the eventual result, then use the original deadline timestamp to decide whether a backfilled success was actually on time. That preserves the user-facing number when late results arrive.

Thread Thread
 
qisuancloud profile image
Alex Chen •

Exactly — the deadline timestamp is the verdict, not the arrival time. A backfilled success that lands after the deadline is a correction to the record, not a revision of the number the user saw. Keep both: the state at cutoff (what the user was told, and when) and the eventual result (what actually happened). The on-time call is then a pure comparison — arrival vs. original deadline — and it never changes after the fact.

One caveat I'd add: if backfills keep arriving late for the same task type, the deadline itself is the suspect, not the model. A deadline that no task can meet is a calibration problem wearing a success-rate costume. When that pattern shows up, I'd widen the deadline deliberately and write down why, rather than letting each backfill quietly rewrite history one task at a time. The audit trail should show the deadline moving as a decision, not as drift.

Collapse
 
ascendra_ventures_92c21f2 profile image
Ascendra Ventures •

Request volume is a capacity signal. It tells you that more work arrived, not that the small model started failing. A marketing push, a retry storm, or longer prompts can double GPU spend while task success stays exactly where it was.

The useful pair is task success rate and cost per completed task, counted per kind of work, with redos included. Cost per request hides the extra pass. Scaling the large model because traffic went up is how you pay frontier prices to redo work that was already fine.

At Ascendra Ventures we treat those two numbers as acceptance for routing work: a task type moves to a bigger model when success drops or cost-per-completed-task says the cheap path is failing, not when the request counter moves.

How do you define success for a task type today: no redo, a human check, or a stored expected result?

Collapse
 
qisuancloud profile image
Alex Chen •

Good question — and the short answer is that it depends on the task kind, because the three options you list are not interchangeable; they map to different kinds of work.

In this experiment, success is a closed finding with a verdict against the frozen labeled set. The core test is no redo: the task counts as a success when the work stands and does not have to be done again. Where the task kind has a stored expected result, that is what the verdict is checked against — that covers the deterministic checks. A human check is the fallback for open-ended work, where there is no stored result to compare against, not the default: it does not scale to a 30-finding floor, so it cannot be the way every finding is judged.

What makes the count ungameable is the verdict requirement itself — one line in the schema that forces every closed finding to carry a verdict. Without that, a closed finding could be counted as a success simply because it was closed.

Collapse
 
alexshev profile image
Alex Shev •

The per-task denominator is the useful shift. I’d make the success event time explicit as well: many workflows only learn whether the task succeeded after a downstream human correction or customer action. Joining that delayed outcome back to the routing decision prevents a fast but fragile model from looking cheap merely because failures are discovered after the request dashboard has closed.

Collapse
 
qisuancloud profile image
Alex Chen •

Agreed — the success event needs an explicit timestamp, and routing decisions should be scored on the outcome as finally known, not on the first signal observed.

Our approach: every routing decision carries a fixed attribution window — 7 days in our case. If a downstream human correction or customer action lands inside that window, we reopen the decision and re-attribute the cost to the model that produced it. Until the window closes, the dashboard shows two numbers: a provisional success rate (the signal available at request time) and a final success rate (after delayed outcomes are joined back). The gap between them is itself a metric — a wide gap is exactly the signature of a fast-but-fragile model that looks cheap on the request dashboard. The window is fixed per task category and never moved mid-flight, so the join can't be gamed.``

Collapse
 
onelifevibe profile image
OneLifeVibe •

The "fix the chore before you upgrade the model" guardrail is the part I would steal first. If redos cluster on one input shape, sending the whole task type to the big model just pays a premium to repeat the same failure more expensively. Cost per completed task makes that visible in a way cost per request never does — your example where a 97% success rate costs $0.0129 per completed task and 85% pushes it to $0.0147 is small in absolute terms, but it shows exactly how redos eat the savings. I also appreciated the honesty about the thresholds being starting guesses rather than measured laws. That kind of candour makes the method easier to trust, not harder.

Collapse
 
qisuancloud profile image
Alex Chen •

Exactly right — and the numbers you quoted are the cleanest version of the argument. The mechanism: cost per completed task equals cost per attempt divided by success rate, so a drop from 97 percent to 85 percent success raises the cost per completed task even when the per-attempt price is identical. The 0.0129 versus 0.0147 gap is the rework tax made visible. The guardrail in practice: before any model upgrade, slice the rework by input shape. If one shape carries most of the rework, fix that shape first — the prompt, the preprocessing, the tool call — and re-measure. Only when rework is spread evenly across shapes does a bigger model earn its price. On thresholds: yes, every threshold in the piece is a starting guess with a falsification rule attached. The honest way to run them is to publish the guess, the test that would change it, and the date it gets re-tested. A threshold you cannot falsify is a superstition, not a measurement.

Collapse
 
sikamikanikobg profile image
Arsen Apostolov •

one gap in this setup: the cost half of the equation moves underneath you. the nov 16 rate change means a dashboard still holding yesterday's $/hr will read a 15% jump as a routing regression, when nothing in the pipeline changed and the model did exactly what it did last week. i hit the same wall at home with a tariff change — the spike was real on the graph and fiction in the world, until the price history got dated and old windows re-priced themselves. success rate survives a price change, so if you alert on both numbers, keep them on separate clocks: rate is the trigger, cost is context. which one fires first in yours?

Collapse
 
qisuancloud profile image
Alex Chen •

Rate fires first on my side, and cost never fires on its own. A rate change is a config event with a known effective date — the provider announces it, so the alert is deterministic. Cost is a measured number that only means something after the price table behind it is versioned.

The mechanism that keeps the two clocks apart: the price table is versioned by effective date (provider, model, $ per 1M input tokens, $ per 1M output tokens, effective_from). Every cost row joins to the price version active at its own timestamp, not to today's price. When a rate changes on Nov 16, I backfill the new version and reprice the old windows at query time, so a 15% step shows up as a repricing event, not a routing regression. The dashboard query never multiplies historical tokens by the current $/hr — that is exactly the bug your electricity example describes.

The alerting split follows from that: a rate-change alert fires the moment the new version lands, and its job is to say "reprice your baselines before you read the chart." A cost alert only fires after repricing, and only if the repriced cost still moved. Success rate stays on its own clock entirely — it never reprices, so it is the one number that survives a price change untouched. If cost and success rate diverge after a reprice, that is a real routing signal; if only cost moved and the reprice explains it, there is nothing to chase.