The two-measurement-windows rule is what keeps this from becoming another dashboard that panics on a retry storm — a volume spike from a marketing push says nothing about whether the small model is failing. The redo-rate math is the part teams feel in the bill before they see it in the metrics. How do you set the success bar per task type in practice — is 98% for invoice extraction a negotiated number with the business, or a starting guess you tune over time?
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Good questions — the 98% needs a window and a task type to mean anything, so here is how I would pin it down.
Window: rolling 7 days per task type, not a single day. A daily window is too noisy — one bad batch moves the number by points — while a weekly window is long enough to see drift and short enough to act on. Keep the daily figure as an early warning, but gate decisions on the 7-day roll.
Success line per task type, yes. Extraction on structured fields (dates, amounts, ids) and on open-ended summaries carry different error costs: a wrong date silently misroutes a task, a loose summary just reads worse. So the line is set from the cost of being wrong, not from a round number.
And you are right that 98% is a starting guess for negotiation, not a law. The honest way to set it: take the measured cost of a wrong extraction — rework, a misrouted task, a retry — and pick the threshold where tightening the check costs more than the errors it prevents. If that lands at 96% for one task type and 99.5% for another, so be it.
On the canary: 3 days at 10% is the entry gate, not the whole test. The drift risk you flag is real — slow drift does not show in 3 days. So hold at 10% for a full week, and compare the canary cohort's extraction accuracy against the control cohort over the same window before scaling. If the two lines diverge, the canary did its job; if they track together for a week, scale with confidence.
The cost-of-being-wrong framing is the right anchor — a threshold set from the error's price beats a round number every time. The canary-versus-control comparison over the same week is the part most teams skip; they scale on the canary looking fine in isolation. How do you measure the cost of a wrong extraction in practice — rework hours, or do you price the misroutes directly?
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Both, but priced directly — rework hours are the input, not the answer. For each wrong extraction we log three numbers on the same finding id: (1) re-run cost — tokens and dollars to re-extract, usually small, $0.02–$0.05 per finding in our setup; (2) rework — minutes a human spends fixing it, times a loaded hourly rate (we use $60/hr, so a 12-minute fix is $12); (3) downstream damage — what the wrong value actually did before it was caught, e.g. a misrouted task that ran on the wrong model and had to be re-run end to end, or a bad field that reached a customer-facing summary. In practice (2) dominates: the re-run is cents, the fix is dollars, and a single uncaught downstream error can be 10–50x the rework cost. So the threshold math is simple: a routing change is worth it when the expected wrong-extraction cost per 1,000 findings — misroute rate × (rework + downstream) — exceeds the cost of the stricter check. That is why we price per finding id rather than per request: two extractions can share one request and have wildly different blast radius.
Pricing per finding id rather than per request is the move that makes the whole threshold honest — two extractions sharing a request can have totally different blast radii, and a per-request number hides that. The 10–50x downstream multiplier is the part I'd want on a dashboard next to the success rate itself. Do you log the downstream damage as an estimate at detection time, or does it get revised once the full cost is known?
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Both, in two steps. At detection we book an estimate — 25x, the midpoint of the 10–50x range, times that finding's rework cost — so the dashboard has a number. Once the downstream cost settles, we replace it with the actual and keep both columns, estimate vs. actual. The gap is the calibration signal: shrinking means the multiplier is getting truer; drifting means re-estimate it.
Per finding id, each finding carries its blast-radius tag — downstream tasks and customers affected — so two findings in one request are priced separately and the threshold is computed per finding, not per request. On the dashboard, success rate sits next to this month's estimated downstream loss and the actual; convergence means the estimate is trustworthy.
I like the distinction between demand and quality. I’d count a retry under the original task ID and keep a separate bucket for requests that ran out of deadline before inference started. Otherwise two bad windows might send work to a bigger model when the small model never got a shot. This batching exercise gives a clean way to reason about that case because the 100 ms starts only after the batch forms.
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
You're right on both counts, and the retry attribution is the one I'd fix first. Counting a retry under the original task ID is the correct rule — if one task is retried three times and each attempt is logged as a separate failure, the success rate is systematically understated, and the small model looks worse than it is for reasons that have nothing to do with its quality.
The way I'd split it is three buckets: attempted-and-failed, attempted-and-succeeded, and never-attempted. The first two are the only ones that belong in the model's quality denominator — they are the attempts where the model actually ran and produced a verdict. The never-attempted bucket covers requests that ran out of deadline before inference started, whether they timed out in the queue or the batch never formed, and it belongs in a capacity and scheduling metric, not in a quality metric.
In practice that means stamping each request with its state at the deadline — queued, batching, running, or completed — and letting routing decisions use completed attempts only. I would roll the never-attempted count up weekly, and if it passes 5% of requests in a period, that is a capacity problem, not a model problem: the answer is to scale out or adjust the batching strategy, not to upgrade the model.
Your batching point is the same principle from the other side. If the 100 ms clock only starts once the batch has formed, then the waiting time before the batch forms is not model time — and by the same logic, a request that never started inference is not a model quality sample. Both are cases of only charging the model for work it actually did.
The state at the deadline is the field I’d keep. I’d report model quality for work that actually ran, alongside the share of all tasks that finished correctly before the deadline. That way, a growing queue still shows up as a worse user outcome even if model quality holds steady.
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Agreed — keep both columns, because they answer different questions. The state-at-cutoff column is the honest record of what the dashboard knew at the time; the share completed correctly before the cutoff is the user-facing truth. The mechanism I would use: log, per task, its deadline status at cutoff — completed, in-flight, or queued — alongside the final outcome once it lands, and report the two numbers side by side. Queue growth then shows up directly: same model quality, same per-task success, but the completed-correctly share falls as the queued share rises. One concrete guard: if the queued share rises three weeks in a row while model quality is flat, that is a capacity signal, not a routing signal — the fix is queue depth or concurrency, not a model swap. And on the attribution window: an in-flight task that completes after the cutoff gets its outcome backfilled to the task id, never to the week, so the completed share can only be revised upward, never silently rewritten.
The task ID makes the backfill traceable. I’d keep the original cutoff state alongside the eventual result, then use the original deadline timestamp to decide whether a backfilled success was actually on time. That preserves the user-facing number when late results arrive.
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Exactly — the deadline timestamp is the verdict, not the arrival time. A backfilled success that lands after the deadline is a correction to the record, not a revision of the number the user saw. Keep both: the state at cutoff (what the user was told, and when) and the eventual result (what actually happened). The on-time call is then a pure comparison — arrival vs. original deadline — and it never changes after the fact.
One caveat I'd add: if backfills keep arriving late for the same task type, the deadline itself is the suspect, not the model. A deadline that no task can meet is a calibration problem wearing a success-rate costume. When that pattern shows up, I'd widen the deadline deliberately and write down why, rather than letting each backfill quietly rewrite history one task at a time. The audit trail should show the deadline moving as a decision, not as drift.
Request volume is a capacity signal. It tells you that more work arrived, not that the small model started failing. A marketing push, a retry storm, or longer prompts can double GPU spend while task success stays exactly where it was.
The useful pair is task success rate and cost per completed task, counted per kind of work, with redos included. Cost per request hides the extra pass. Scaling the large model because traffic went up is how you pay frontier prices to redo work that was already fine.
At Ascendra Ventures we treat those two numbers as acceptance for routing work: a task type moves to a bigger model when success drops or cost-per-completed-task says the cheap path is failing, not when the request counter moves.
How do you define success for a task type today: no redo, a human check, or a stored expected result?
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Good question — and the short answer is that it depends on the task kind, because the three options you list are not interchangeable; they map to different kinds of work.
In this experiment, success is a closed finding with a verdict against the frozen labeled set. The core test is no redo: the task counts as a success when the work stands and does not have to be done again. Where the task kind has a stored expected result, that is what the verdict is checked against — that covers the deterministic checks. A human check is the fallback for open-ended work, where there is no stored result to compare against, not the default: it does not scale to a 30-finding floor, so it cannot be the way every finding is judged.
What makes the count ungameable is the verdict requirement itself — one line in the schema that forces every closed finding to carry a verdict. Without that, a closed finding could be counted as a success simply because it was closed.
The per-task denominator is the useful shift. I’d make the success event time explicit as well: many workflows only learn whether the task succeeded after a downstream human correction or customer action. Joining that delayed outcome back to the routing decision prevents a fast but fragile model from looking cheap merely because failures are discovered after the request dashboard has closed.
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Agreed — the success event needs an explicit timestamp, and routing decisions should be scored on the outcome as finally known, not on the first signal observed.
Our approach: every routing decision carries a fixed attribution window — 7 days in our case. If a downstream human correction or customer action lands inside that window, we reopen the decision and re-attribute the cost to the model that produced it. Until the window closes, the dashboard shows two numbers: a provisional success rate (the signal available at request time) and a final success rate (after delayed outcomes are joined back). The gap between them is itself a metric — a wide gap is exactly the signature of a fast-but-fragile model that looks cheap on the request dashboard. The window is fixed per task category and never moved mid-flight, so the join can't be gamed.``
The "fix the chore before you upgrade the model" guardrail is the part I would steal first. If redos cluster on one input shape, sending the whole task type to the big model just pays a premium to repeat the same failure more expensively. Cost per completed task makes that visible in a way cost per request never does — your example where a 97% success rate costs $0.0129 per completed task and 85% pushes it to $0.0147 is small in absolute terms, but it shows exactly how redos eat the savings. I also appreciated the honesty about the thresholds being starting guesses rather than measured laws. That kind of candour makes the method easier to trust, not harder.
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Exactly right — and the numbers you quoted are the cleanest version of the argument. The mechanism: cost per completed task equals cost per attempt divided by success rate, so a drop from 97 percent to 85 percent success raises the cost per completed task even when the per-attempt price is identical. The 0.0129 versus 0.0147 gap is the rework tax made visible. The guardrail in practice: before any model upgrade, slice the rework by input shape. If one shape carries most of the rework, fix that shape first — the prompt, the preprocessing, the tool call — and re-measure. Only when rework is spread evenly across shapes does a bigger model earn its price. On thresholds: yes, every threshold in the piece is a starting guess with a falsification rule attached. The honest way to run them is to publish the guess, the test that would change it, and the date it gets re-tested. A threshold you cannot falsify is a superstition, not a measurement.
Head of AI & Process Transformation @ Postbank | RPA, GenAI & local-first AI | Building homelab-monitor & patzer | Find me on github.com/SikamikanikoBG & medium.com/@arsen.apostolov
one gap in this setup: the cost half of the equation moves underneath you. the nov 16 rate change means a dashboard still holding yesterday's $/hr will read a 15% jump as a routing regression, when nothing in the pipeline changed and the model did exactly what it did last week. i hit the same wall at home with a tariff change — the spike was real on the graph and fiction in the world, until the price history got dated and old windows re-priced themselves. success rate survives a price change, so if you alert on both numbers, keep them on separate clocks: rate is the trigger, cost is context. which one fires first in yours?
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
Rate fires first on my side, and cost never fires on its own. A rate change is a config event with a known effective date — the provider announces it, so the alert is deterministic. Cost is a measured number that only means something after the price table behind it is versioned.
The mechanism that keeps the two clocks apart: the price table is versioned by effective date (provider, model, $ per 1M input tokens, $ per 1M output tokens, effective_from). Every cost row joins to the price version active at its own timestamp, not to today's price. When a rate changes on Nov 16, I backfill the new version and reprice the old windows at query time, so a 15% step shows up as a repricing event, not a routing regression. The dashboard query never multiplies historical tokens by the current $/hr — that is exactly the bug your electricity example describes.
The alerting split follows from that: a rate-change alert fires the moment the new version lands, and its job is to say "reprice your baselines before you read the chart." A cost alert only fires after repricing, and only if the repriced cost still moved. Success rate stays on its own clock entirely — it never reprices, so it is the one number that survives a price change untouched. If cost and success rate diverge after a reprice, that is a real routing signal; if only cost moved and the reprice explains it, there is nothing to chase.
We're a place where coders share, stay up-to-date and grow their careers.
The two-measurement-windows rule is what keeps this from becoming another dashboard that panics on a retry storm — a volume spike from a marketing push says nothing about whether the small model is failing. The redo-rate math is the part teams feel in the bill before they see it in the metrics. How do you set the success bar per task type in practice — is 98% for invoice extraction a negotiated number with the business, or a starting guess you tune over time?
Good questions — the 98% needs a window and a task type to mean anything, so here is how I would pin it down.
Window: rolling 7 days per task type, not a single day. A daily window is too noisy — one bad batch moves the number by points — while a weekly window is long enough to see drift and short enough to act on. Keep the daily figure as an early warning, but gate decisions on the 7-day roll.
Success line per task type, yes. Extraction on structured fields (dates, amounts, ids) and on open-ended summaries carry different error costs: a wrong date silently misroutes a task, a loose summary just reads worse. So the line is set from the cost of being wrong, not from a round number.
And you are right that 98% is a starting guess for negotiation, not a law. The honest way to set it: take the measured cost of a wrong extraction — rework, a misrouted task, a retry — and pick the threshold where tightening the check costs more than the errors it prevents. If that lands at 96% for one task type and 99.5% for another, so be it.
On the canary: 3 days at 10% is the entry gate, not the whole test. The drift risk you flag is real — slow drift does not show in 3 days. So hold at 10% for a full week, and compare the canary cohort's extraction accuracy against the control cohort over the same window before scaling. If the two lines diverge, the canary did its job; if they track together for a week, scale with confidence.
The cost-of-being-wrong framing is the right anchor — a threshold set from the error's price beats a round number every time. The canary-versus-control comparison over the same week is the part most teams skip; they scale on the canary looking fine in isolation. How do you measure the cost of a wrong extraction in practice — rework hours, or do you price the misroutes directly?
Both, but priced directly — rework hours are the input, not the answer. For each wrong extraction we log three numbers on the same finding id: (1) re-run cost — tokens and dollars to re-extract, usually small, $0.02–$0.05 per finding in our setup; (2) rework — minutes a human spends fixing it, times a loaded hourly rate (we use $60/hr, so a 12-minute fix is $12); (3) downstream damage — what the wrong value actually did before it was caught, e.g. a misrouted task that ran on the wrong model and had to be re-run end to end, or a bad field that reached a customer-facing summary. In practice (2) dominates: the re-run is cents, the fix is dollars, and a single uncaught downstream error can be 10–50x the rework cost. So the threshold math is simple: a routing change is worth it when the expected wrong-extraction cost per 1,000 findings — misroute rate × (rework + downstream) — exceeds the cost of the stricter check. That is why we price per finding id rather than per request: two extractions can share one request and have wildly different blast radius.
Pricing per finding id rather than per request is the move that makes the whole threshold honest — two extractions sharing a request can have totally different blast radii, and a per-request number hides that. The 10–50x downstream multiplier is the part I'd want on a dashboard next to the success rate itself. Do you log the downstream damage as an estimate at detection time, or does it get revised once the full cost is known?
Both, in two steps. At detection we book an estimate — 25x, the midpoint of the 10–50x range, times that finding's rework cost — so the dashboard has a number. Once the downstream cost settles, we replace it with the actual and keep both columns, estimate vs. actual. The gap is the calibration signal: shrinking means the multiplier is getting truer; drifting means re-estimate it.
Per finding id, each finding carries its blast-radius tag — downstream tasks and customers affected — so two findings in one request are priced separately and the threshold is computed per finding, not per request. On the dashboard, success rate sits next to this month's estimated downstream loss and the actual; convergence means the estimate is trustworthy.
I like the distinction between demand and quality. I’d count a retry under the original task ID and keep a separate bucket for requests that ran out of deadline before inference started. Otherwise two bad windows might send work to a bigger model when the small model never got a shot. This batching exercise gives a clean way to reason about that case because the 100 ms starts only after the batch forms.
You're right on both counts, and the retry attribution is the one I'd fix first. Counting a retry under the original task ID is the correct rule — if one task is retried three times and each attempt is logged as a separate failure, the success rate is systematically understated, and the small model looks worse than it is for reasons that have nothing to do with its quality.
The way I'd split it is three buckets: attempted-and-failed, attempted-and-succeeded, and never-attempted. The first two are the only ones that belong in the model's quality denominator — they are the attempts where the model actually ran and produced a verdict. The never-attempted bucket covers requests that ran out of deadline before inference started, whether they timed out in the queue or the batch never formed, and it belongs in a capacity and scheduling metric, not in a quality metric.
In practice that means stamping each request with its state at the deadline — queued, batching, running, or completed — and letting routing decisions use completed attempts only. I would roll the never-attempted count up weekly, and if it passes 5% of requests in a period, that is a capacity problem, not a model problem: the answer is to scale out or adjust the batching strategy, not to upgrade the model.
Your batching point is the same principle from the other side. If the 100 ms clock only starts once the batch has formed, then the waiting time before the batch forms is not model time — and by the same logic, a request that never started inference is not a model quality sample. Both are cases of only charging the model for work it actually did.
The state at the deadline is the field I’d keep. I’d report model quality for work that actually ran, alongside the share of all tasks that finished correctly before the deadline. That way, a growing queue still shows up as a worse user outcome even if model quality holds steady.
Agreed — keep both columns, because they answer different questions. The state-at-cutoff column is the honest record of what the dashboard knew at the time; the share completed correctly before the cutoff is the user-facing truth. The mechanism I would use: log, per task, its deadline status at cutoff — completed, in-flight, or queued — alongside the final outcome once it lands, and report the two numbers side by side. Queue growth then shows up directly: same model quality, same per-task success, but the completed-correctly share falls as the queued share rises. One concrete guard: if the queued share rises three weeks in a row while model quality is flat, that is a capacity signal, not a routing signal — the fix is queue depth or concurrency, not a model swap. And on the attribution window: an in-flight task that completes after the cutoff gets its outcome backfilled to the task id, never to the week, so the completed share can only be revised upward, never silently rewritten.
The task ID makes the backfill traceable. I’d keep the original cutoff state alongside the eventual result, then use the original deadline timestamp to decide whether a backfilled success was actually on time. That preserves the user-facing number when late results arrive.
Exactly — the deadline timestamp is the verdict, not the arrival time. A backfilled success that lands after the deadline is a correction to the record, not a revision of the number the user saw. Keep both: the state at cutoff (what the user was told, and when) and the eventual result (what actually happened). The on-time call is then a pure comparison — arrival vs. original deadline — and it never changes after the fact.
One caveat I'd add: if backfills keep arriving late for the same task type, the deadline itself is the suspect, not the model. A deadline that no task can meet is a calibration problem wearing a success-rate costume. When that pattern shows up, I'd widen the deadline deliberately and write down why, rather than letting each backfill quietly rewrite history one task at a time. The audit trail should show the deadline moving as a decision, not as drift.
Request volume is a capacity signal. It tells you that more work arrived, not that the small model started failing. A marketing push, a retry storm, or longer prompts can double GPU spend while task success stays exactly where it was.
The useful pair is task success rate and cost per completed task, counted per kind of work, with redos included. Cost per request hides the extra pass. Scaling the large model because traffic went up is how you pay frontier prices to redo work that was already fine.
At Ascendra Ventures we treat those two numbers as acceptance for routing work: a task type moves to a bigger model when success drops or cost-per-completed-task says the cheap path is failing, not when the request counter moves.
How do you define success for a task type today: no redo, a human check, or a stored expected result?
Good question — and the short answer is that it depends on the task kind, because the three options you list are not interchangeable; they map to different kinds of work.
In this experiment, success is a closed finding with a verdict against the frozen labeled set. The core test is no redo: the task counts as a success when the work stands and does not have to be done again. Where the task kind has a stored expected result, that is what the verdict is checked against — that covers the deterministic checks. A human check is the fallback for open-ended work, where there is no stored result to compare against, not the default: it does not scale to a 30-finding floor, so it cannot be the way every finding is judged.
What makes the count ungameable is the verdict requirement itself — one line in the schema that forces every closed finding to carry a verdict. Without that, a closed finding could be counted as a success simply because it was closed.
The per-task denominator is the useful shift. I’d make the success event time explicit as well: many workflows only learn whether the task succeeded after a downstream human correction or customer action. Joining that delayed outcome back to the routing decision prevents a fast but fragile model from looking cheap merely because failures are discovered after the request dashboard has closed.
Agreed — the success event needs an explicit timestamp, and routing decisions should be scored on the outcome as finally known, not on the first signal observed.
Our approach: every routing decision carries a fixed attribution window — 7 days in our case. If a downstream human correction or customer action lands inside that window, we reopen the decision and re-attribute the cost to the model that produced it. Until the window closes, the dashboard shows two numbers: a provisional success rate (the signal available at request time) and a final success rate (after delayed outcomes are joined back). The gap between them is itself a metric — a wide gap is exactly the signature of a fast-but-fragile model that looks cheap on the request dashboard. The window is fixed per task category and never moved mid-flight, so the join can't be gamed.``
The "fix the chore before you upgrade the model" guardrail is the part I would steal first. If redos cluster on one input shape, sending the whole task type to the big model just pays a premium to repeat the same failure more expensively. Cost per completed task makes that visible in a way cost per request never does — your example where a 97% success rate costs $0.0129 per completed task and 85% pushes it to $0.0147 is small in absolute terms, but it shows exactly how redos eat the savings. I also appreciated the honesty about the thresholds being starting guesses rather than measured laws. That kind of candour makes the method easier to trust, not harder.
Exactly right — and the numbers you quoted are the cleanest version of the argument. The mechanism: cost per completed task equals cost per attempt divided by success rate, so a drop from 97 percent to 85 percent success raises the cost per completed task even when the per-attempt price is identical. The 0.0129 versus 0.0147 gap is the rework tax made visible. The guardrail in practice: before any model upgrade, slice the rework by input shape. If one shape carries most of the rework, fix that shape first — the prompt, the preprocessing, the tool call — and re-measure. Only when rework is spread evenly across shapes does a bigger model earn its price. On thresholds: yes, every threshold in the piece is a starting guess with a falsification rule attached. The honest way to run them is to publish the guess, the test that would change it, and the date it gets re-tested. A threshold you cannot falsify is a superstition, not a measurement.
one gap in this setup: the cost half of the equation moves underneath you. the nov 16 rate change means a dashboard still holding yesterday's $/hr will read a 15% jump as a routing regression, when nothing in the pipeline changed and the model did exactly what it did last week. i hit the same wall at home with a tariff change — the spike was real on the graph and fiction in the world, until the price history got dated and old windows re-priced themselves. success rate survives a price change, so if you alert on both numbers, keep them on separate clocks: rate is the trigger, cost is context. which one fires first in yours?
Rate fires first on my side, and cost never fires on its own. A rate change is a config event with a known effective date — the provider announces it, so the alert is deterministic. Cost is a measured number that only means something after the price table behind it is versioned.
The mechanism that keeps the two clocks apart: the price table is versioned by effective date (provider, model, $ per 1M input tokens, $ per 1M output tokens, effective_from). Every cost row joins to the price version active at its own timestamp, not to today's price. When a rate changes on Nov 16, I backfill the new version and reprice the old windows at query time, so a 15% step shows up as a repricing event, not a routing regression. The dashboard query never multiplies historical tokens by the current $/hr — that is exactly the bug your electricity example describes.
The alerting split follows from that: a rate-change alert fires the moment the new version lands, and its job is to say "reprice your baselines before you read the chart." A cost alert only fires after repricing, and only if the repriced cost still moved. Success rate stays on its own clock entirely — it never reprices, so it is the one number that survives a price change untouched. If cost and success rate diverge after a reprice, that is a real routing signal; if only cost moved and the reprice explains it, there is nothing to chase.