The "series that does not exist looks healthy" failure mode is the one that keeps biting monitoring setups. We've started pairing every alert expression with a second check that the metric family is actually being scraped. If scrape samples for that job drop to zero, that pages separately from the alert itself. Otherwise you get exactly this: a correct rule watching nothing, and a green dashboard that means "unobserved," not "fine."
Thanks, and "unobserved, not fine" I think is the right way to put it. I don't have this yet: no alert in the repo watches up or scrape_samples_scraped.
Checking it against Part 4, though, a job-level check wouldn't have caught that bug. Transform was being scraped fine, with 28 metric families. The one missing was pybreaker_state. So I think it needs both: up == 0 for a dead target, and absent() on each series a rule depends on, as Amorizz suggested on this thread. You mentioned checking the metric family. Do you do that per family, or does the job-level sample count cover it for you?
One more gap: my alerts don't page anyone yet. There's no Alertmanager in the stack, and Part 5's security doc now says so.
absent() closes the gap where the series never existed, but it still only proves the rule can fire — not that firing reaches anyone. One step up the chain the notification path can die just as quietly: routing drift, a channel that got retired, a mute window somebody left behind. From the rule's side those all look identical to a healthy system.
The cheap way to know somebody is actually watching is to schedule the failure yourself: a cron job that flips pybreaker_state{name="pds"} to 1 for one scrape interval, plus a dead-man switch that pages when the end-to-end ack — alert fired, notification delivered, receipt logged — does not arrive within five minutes. Once a shift is enough. You stop asserting the alerting chain works and start measuring it, which is the same move you made on the pipeline side.
Thanks Nan @eye_java_420f1faa10ae8b86 , you've found a real gap, and it's bigger than the one you describe. I checked after reading this. In the demo stack, the notification path didn't decay because it was never connected. prometheus.yml has no alerting block, and Compose doesn't run Alertmanager. So when PDSCircuitOpen went Firing in Part 4, it reached nobody.
That's the same mistake as the alert with no series, one step further along. I proved the rule fires and treated that as proof someone gets told.
The fix is to wire Alertmanager in first. Then I'd add an always-firing watchdog alert routed to a dead-man switch, so I'm measuring delivery, not just assuming it. Your scheduled real failure goes further, because it tests the rule as well as the route. I'll write this up.
That test case named "with no series exported the rule is silent" is the one I'd frame. A cheap companion is a rule that fires on the absence itself: absent(pybreaker_state{name="pds"}) with a for: 10m, so a renamed metric or a dropped exporter becomes its own alert instead of looking like calm. The for matters, because a fresh deploy can lack the series for a scrape or two. Error trackers have the same blind spot, where zero events from a service looks like health. Would you generate those absent rules from alerts.yml, or keep them by hand?
Thanks, @amorizz, that's a good addition, and I'm taking it. An absent() rule would have caught the original bug on day one because the series never existed. You're right that the for matters too.
I'd keep them by hand for now. Generating them from alerts.yml sounds tidy, but it breaks on labelled counters. hl7_field_dropped_total and hl7_dlq_total have no series until the first drop or dead letter, so a generated absent rule would fire on a healthy, quiet service. pybreaker_state is set when the service starts, so that one is safe. I'll add it along with a promtool case that proves it fires.
I research AI tools, APIs & agents — comparing specs, pricing and real user reports — then publish honest rankings so you don't waste money. Research-first, never sponsored fluff.
amorizz's absent() companion rule is the bit I'd frame — a rule that fires on the absence itself, so a renamed metric or a dropped exporter can't look healthy. The getattr default one is nastier though: a gauge sitting at closed while the breaker is open is worse than no metric at all, because now there's a panel to trust. Did that one come from a real incident, or did the test catch it before it shipped?
Thanks, @paullovelace Paul. No incident. This is a sandbox project, and nothing runs in production, so the test caught it before it shipped. That's what bothers me about it, too. Without that test, I'd have merged it and been pleased with the new panel.
The async decorator bug was worse in that respect. The unit test passed, and only make break-pds against the real stack showed it.
"You cannot tell the difference between nothing being wrong and nothing being watched" is going on a sticky note. An alert rule pointing at a series nobody exports looks identical to a healthy system. Have you thought about a meta-check that fails CI when an alert's expression returns no series at all against a scrape of the running services?
I write when an idea won't leave me alone 🧠 Building AI agents and the tools to build AI agents. Love connecting AI with other fields and yapping about all of them.
A decorator that records a success because it got a coroutine back is nasty, and 16 PDS failures with zero counted shows how quiet it was. I would have trusted that sync unit test too. Is make break-pds headed for CI next to promtool, or does it need too much of the stack?
Thanks, @kartik-nvjk Kartik. Same here; I trusted that sync test until the running stack showed 16 failures and none counted.
It isn't in CI yet, but it should be. GitHub's Ubuntu runners have Docker, so the Compose stack fits. The CI version would read the gauge straight from transform's /metrics and check it reaches 1, rather than wait the two minutes for Prometheus to show Firing. That keeps the job short and still tests the async path that broke.
We're a place where coders share, stay up-to-date and grow their careers.
The "series that does not exist looks healthy" failure mode is the one that keeps biting monitoring setups. We've started pairing every alert expression with a second check that the metric family is actually being scraped. If scrape samples for that job drop to zero, that pages separately from the alert itself. Otherwise you get exactly this: a correct rule watching nothing, and a green dashboard that means "unobserved," not "fine."
Thanks, and "unobserved, not fine" I think is the right way to put it. I don't have this yet: no alert in the repo watches up or scrape_samples_scraped.
Checking it against Part 4, though, a job-level check wouldn't have caught that bug. Transform was being scraped fine, with 28 metric families. The one missing was pybreaker_state. So I think it needs both: up == 0 for a dead target, and absent() on each series a rule depends on, as Amorizz suggested on this thread. You mentioned checking the metric family. Do you do that per family, or does the job-level sample count cover it for you?
One more gap: my alerts don't page anyone yet. There's no Alertmanager in the stack, and Part 5's security doc now says so.
absent() closes the gap where the series never existed, but it still only proves the rule can fire — not that firing reaches anyone. One step up the chain the notification path can die just as quietly: routing drift, a channel that got retired, a mute window somebody left behind. From the rule's side those all look identical to a healthy system.
The cheap way to know somebody is actually watching is to schedule the failure yourself: a cron job that flips
pybreaker_state{name="pds"}to 1 for one scrape interval, plus a dead-man switch that pages when the end-to-end ack — alert fired, notification delivered, receipt logged — does not arrive within five minutes. Once a shift is enough. You stop asserting the alerting chain works and start measuring it, which is the same move you made on the pipeline side.Thanks Nan @eye_java_420f1faa10ae8b86 , you've found a real gap, and it's bigger than the one you describe. I checked after reading this. In the demo stack, the notification path didn't decay because it was never connected. prometheus.yml has no alerting block, and Compose doesn't run Alertmanager. So when PDSCircuitOpen went Firing in Part 4, it reached nobody.
That's the same mistake as the alert with no series, one step further along. I proved the rule fires and treated that as proof someone gets told.
The fix is to wire Alertmanager in first. Then I'd add an always-firing watchdog alert routed to a dead-man switch, so I'm measuring delivery, not just assuming it. Your scheduled real failure goes further, because it tests the rule as well as the route. I'll write this up.
That test case named "with no series exported the rule is silent" is the one I'd frame. A cheap companion is a rule that fires on the absence itself:
absent(pybreaker_state{name="pds"})with afor: 10m, so a renamed metric or a dropped exporter becomes its own alert instead of looking like calm. Theformatters, because a fresh deploy can lack the series for a scrape or two. Error trackers have the same blind spot, where zero events from a service looks like health. Would you generate those absent rules from alerts.yml, or keep them by hand?Thanks, @amorizz, that's a good addition, and I'm taking it. An absent() rule would have caught the original bug on day one because the series never existed. You're right that the for matters too.
I'd keep them by hand for now. Generating them from alerts.yml sounds tidy, but it breaks on labelled counters. hl7_field_dropped_total and hl7_dlq_total have no series until the first drop or dead letter, so a generated absent rule would fire on a healthy, quiet service. pybreaker_state is set when the service starts, so that one is safe. I'll add it along with a promtool case that proves it fires.
amorizz's absent() companion rule is the bit I'd frame — a rule that fires on the absence itself, so a renamed metric or a dropped exporter can't look healthy. The getattr default one is nastier though: a gauge sitting at closed while the breaker is open is worse than no metric at all, because now there's a panel to trust. Did that one come from a real incident, or did the test catch it before it shipped?
Thanks, @paullovelace Paul. No incident. This is a sandbox project, and nothing runs in production, so the test caught it before it shipped. That's what bothers me about it, too. Without that test, I'd have merged it and been pleased with the new panel.
The async decorator bug was worse in that respect. The unit test passed, and only make break-pds against the real stack showed it.
"You cannot tell the difference between nothing being wrong and nothing being watched" is going on a sticky note. An alert rule pointing at a series nobody exports looks identical to a healthy system. Have you thought about a meta-check that fails CI when an alert's expression returns no series at all against a scrape of the running services?
A decorator that records a success because it got a coroutine back is nasty, and 16 PDS failures with zero counted shows how quiet it was. I would have trusted that sync unit test too. Is make break-pds headed for CI next to promtool, or does it need too much of the stack?
Thanks, @kartik-nvjk Kartik. Same here; I trusted that sync test until the running stack showed 16 failures and none counted.
It isn't in CI yet, but it should be. GitHub's Ubuntu runners have Docker, so the Compose stack fits. The CI version would read the gauge straight from transform's /metrics and check it reaches 1, rather than wait the two minutes for Prometheus to show Firing. That keeps the job short and still tests the async path that broke.