Senior Cloud and AI Engineer in Dallas, TX. 8+ years across Azure, GCP, and AWS. Leading API architecture currently. I write about LLM platforms, agentic AI, and production ML systems.
The step and prompt_version labels are the detail that would have saved the two hours. We see the same shape from the gateway side: attribution per workflow step turns a bill spike into one query. The retry cap matters too, but without labels you do not even know which loop to cap.
I test what ships and publish what I find, failures included.
Kubernetes, AWS and supply chain security.
Docker Captain, CNCF Ambassador, IBM Champion. 1.8M+ Docker Hub pulls.
Labels fix attribution only as long as every caller sets them correctly. A prompt_version label is something a service writes about itself, and nine services on one service account give the bill exactly one identity to report.
In the air-gapped agent cluster I run, every caller shows a Kubernetes service account token for the gateway's audience, checked offline against the cluster's own keys, and quotas follow the token's subject, not a header. A daily gate also fails the run unless the agent's own request count equals the gateway's.
With one identity per workload, the 07:40 question is one query. With one shared account, it stays a Slack thread.
ML engineer focused on cost-efficient AI infrastructure. I benchmark GPU clouds and share practical guides for running LLMs in production.
📬 Telegram: https://t.me/qisuancloud
The distinction between attribution and prevention is the part that needs more emphasis: labels would have located the loop faster, but they would not have stopped it. Would a retry cap that also shrinks the carried context on each rejection have contained the cost growth more directly than a fixed attempt limit? A cap on cumulative tokens per workflow run, rather than attempts alone, seems like the missing boundary condition here.
GMT strategy and custom B2B outreach lists on request: people and companies by ICP, domain, profile URL, email or name. 1B+ profiles. Claude Code plugins, API, MCP. [email protected]
A cumulative token cap per run is the one I would pick too, because it holds however the loop is shaped. I would check it before each retry rather than alert on it: the loop reads the remaining budget and stops itself at attempt 3, instead of someone finding out at 07:40.
We ended up making the ticket the label. Each Claude Code session works one ticket, and the agent pushes the cost of each turn onto that ticket. A loop like your v4 reviewer would show up as one ticket with a long run of turns while it runs, not as a project total the next morning. Your caveat holds for us too: that number is self-reported by the agent. Do you reconcile labeled spend against the billing export, and how far apart do they land?
Nine services on one service account explains most of the incident. What works for us is a record per call that names the caller and the cost, written at the moment of the call, because labels added afterwards rarely line up with the spike. Did you end up splitting the service accounts, or adding a caller label to each request?
Cost incidents are usually ownership incidents before they are model incidents. A practical guardrail is to require every workload to carry an owner, budget, environment, and kill switch, then make spend visible at the same level people use to make deployment decisions.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
The bit that stuck with me was how ordinary the failure was: a stricter reviewer in prompt v4, a loop with no ceiling, and then eight hours before anyone noticed because billing was the only signal. I keep thinking about your AI-7 point — tokens-per-minute by use_case would have caught that at 23:30, not 07:40. Do you find teams actually keep that alert tuned, or does it go noisy once a legit batch job runs? And on the taxonomy, the 'about 100 values' rule for labels is a neat guardrail; I'd worry prompt_version alone could explode if teams start stamping every experiment, so how strict are you about what counts as a version?
I write about Next.js + TypeScript + AI engineering, with a focus on making sites discoverable to LLMs and AI search. Long-form notes at mudassirkhan.me
the retry loop with growing context is the part that stings. each rewrite carries the full ticket history plus the previous draft, so cost grows closer to quadratic than linear with every round. we got burned by the same shape and what actually fixed it was a hard cap on rewrite rounds plus logging tokens per round, so a runaway loop shows up in minutes instead of in the next morning billing alert. labels tell you who, but only live token metrics tell you while it is happening. do you emit token counts per workflow step now, or are you still leaning on the billing export?
I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
v4 went out at 16:30 and the first thing to actually run it was the 23:00 nightly job, which is when the rejections started. I replay prompt changes against a slice of past cases before merge, and for a review step the number I watch is rejections per draft. Did a pre-merge replay come up when you were writing AI-1 to AI-8?
One label I'd add next to step and prompt_version: cache-hit tokens per call. In a loop like that rewrite step, every retry resends the full ticket history, so the bill depends heavily on whether that prefix hits the provider's prompt cache. Cached-input prices vary a lot even for the same open model; for GLM-5.3 I see roughly $0.07 to $0.26 per 1M cached tokens across hosts, versus $0.50 to $1.40 uncached on fp8 and native hosts (list prices checked 2026-10-27). If the label shows the cache-hit share collapsing, for example because the previous draft gets inserted before the stable history, you can cut the spike by reordering the prompt without touching the retry logic.
The step and prompt_version labels are the detail that would have saved the two hours. We see the same shape from the gateway side: attribution per workflow step turns a bill spike into one query. The retry cap matters too, but without labels you do not even know which loop to cap.
Labels fix attribution only as long as every caller sets them correctly. A
prompt_versionlabel is something a service writes about itself, and nine services on one service account give the bill exactly one identity to report.In the air-gapped agent cluster I run, every caller shows a Kubernetes service account token for the gateway's audience, checked offline against the cluster's own keys, and quotas follow the token's subject, not a header. A daily gate also fails the run unless the agent's own request count equals the gateway's.
With one identity per workload, the 07:40 question is one query. With one shared account, it stays a Slack thread.
The distinction between attribution and prevention is the part that needs more emphasis: labels would have located the loop faster, but they would not have stopped it. Would a retry cap that also shrinks the carried context on each rejection have contained the cost growth more directly than a fixed attempt limit? A cap on cumulative tokens per workflow run, rather than attempts alone, seems like the missing boundary condition here.
A cumulative token cap per run is the one I would pick too, because it holds however the loop is shaped. I would check it before each retry rather than alert on it: the loop reads the remaining budget and stops itself at attempt 3, instead of someone finding out at 07:40.
We ended up making the ticket the label. Each Claude Code session works one ticket, and the agent pushes the cost of each turn onto that ticket. A loop like your v4 reviewer would show up as one ticket with a long run of turns while it runs, not as a project total the next morning. Your caveat holds for us too: that number is self-reported by the agent. Do you reconcile labeled spend against the billing export, and how far apart do they land?
Nine services on one service account explains most of the incident. What works for us is a record per call that names the caller and the cost, written at the moment of the call, because labels added afterwards rarely line up with the spike. Did you end up splitting the service accounts, or adding a caller label to each request?
Cost incidents are usually ownership incidents before they are model incidents. A practical guardrail is to require every workload to carry an owner, budget, environment, and kill switch, then make spend visible at the same level people use to make deployment decisions.
The bit that stuck with me was how ordinary the failure was: a stricter reviewer in prompt v4, a loop with no ceiling, and then eight hours before anyone noticed because billing was the only signal. I keep thinking about your AI-7 point — tokens-per-minute by use_case would have caught that at 23:30, not 07:40. Do you find teams actually keep that alert tuned, or does it go noisy once a legit batch job runs? And on the taxonomy, the 'about 100 values' rule for labels is a neat guardrail; I'd worry prompt_version alone could explode if teams start stamping every experiment, so how strict are you about what counts as a version?
the retry loop with growing context is the part that stings. each rewrite carries the full ticket history plus the previous draft, so cost grows closer to quadratic than linear with every round. we got burned by the same shape and what actually fixed it was a hard cap on rewrite rounds plus logging tokens per round, so a runaway loop shows up in minutes instead of in the next morning billing alert. labels tell you who, but only live token metrics tell you while it is happening. do you emit token counts per workflow step now, or are you still leaning on the billing export?
v4 went out at 16:30 and the first thing to actually run it was the 23:00 nightly job, which is when the rejections started. I replay prompt changes against a slice of past cases before merge, and for a review step the number I watch is rejections per draft. Did a pre-merge replay come up when you were writing AI-1 to AI-8?
One label I'd add next to step and prompt_version: cache-hit tokens per call. In a loop like that rewrite step, every retry resends the full ticket history, so the bill depends heavily on whether that prefix hits the provider's prompt cache. Cached-input prices vary a lot even for the same open model; for GLM-5.3 I see roughly $0.07 to $0.26 per 1M cached tokens across hosts, versus $0.50 to $1.40 uncached on fp8 and native hosts (list prices checked 2026-10-27). If the label shows the cache-hit share collapsing, for example because the previous draft gets inserted before the stable history, you can cut the spike by reordering the prompt without touching the retry logic.
an unlabeled retry loop is a quiet way to burn money.
An AI bill with no clear owner sounds like a mess.