A backup script on one of my boxes ran every night for three weeks. It exited 0. It wrote nothing. Nothing alerted, and from the system's perspective, nothing had failed.
I found out because I needed a file from six months ago.
This is not an unusual story. If you have run servers long enough, you have your own version of it. What made it interesting to me was how many different ways the same class of failure can hide, and how little of it is covered by the monitoring people typically install.
The failure that doesn't look like a failure
The mental model most of us carry is: backup → error → alert. But the failures that actually bite are the ones where the exit code is zero.
- The target path moved because a volume got remounted somewhere else, so
rsynccheerfully copied nothing. - The destination filled up, and the tool wrote a zero-byte file and called it a day.
- A
cronentry was dropped during an OS upgrade. The job did not fail — it stopped existing. - The script succeeded, but the snapshot retention logic deleted the new snapshot.
-
systemdput the unit into a failed state weeks ago and nobody looked.
In none of these does the thing you are monitoring — "did the process exit non-zero" — fire.
So the question has to change shape. Instead of "did the job succeed", ask "how old is the newest artifact". That question is answerable without trusting the job at all:
find /backups -maxdepth 1 -type f -printf '%T@ %p\n' | sort -n | tail -1
If the newest file is older than your RPO, you are in trouble regardless of what any exit code said. That one inversion is most of the value.
The second thing I got wrong: df -h
df -h said 40% free. The application still could not write, because inodes were exhausted — a mail queue had accumulated a few million tiny files.
df -h and df -i answer different questions, and the second one fails harder when it fails:
df -P -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 > 85'
df -Pi -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0 > 85'
A related trap: percent-used is the wrong metric for alerting. A volume growing at 2 GB/hour is an emergency at 60%; one sitting at 84% for two years is not. What you want is time to full:
time_to_full ≈ (size_bytes − used_bytes) / growth_bytes_per_hour
That requires keeping a little history, which is exactly what people skip because the dashboard already "shows disk".
Clock skew, the silent one
This one surprised me. A box's clock drifted a few minutes. Nothing crashed. But:
- TLS validation against external endpoints began failing intermittently
-
cronfired at surprising times, because it works off wall-clock time - Correlating logs across machines became fiction
timedatectl will tell you whether the clock is synchronised. One line, almost nobody has the check, and when it is wrong it makes every other investigation harder.
systemctl --failed is criminally underused
$ systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● [email protected] loaded failed failed Daily backup
A unit can sit in failed for months. It is not in your logs unless you are looking, and if it is the backup unit, the backup has been dead the whole time while the timer rescheduled itself every night.
What I ended up building
I wanted one command that answered: is anything about to break, and did last night's backup actually run? Ten read-only checks, aggregated:
$ vaultcheck audit
Vaultcheck audit — web-01 — overall: CRIT
CRIT backup newest artifact is 22d 4h old (threshold 26h)
WARN disk /var at 87% (5.2 GB free)
OK inodes max 31% (/)
OK memory 1.4 GB / 8.0 GB used
OK load 0.42 / 0.31 / 0.28
CRIT systemd [email protected] failed
OK cron crontab present, 6 jobs
WARN clock not synchronised (drift ~184s)
OK logs log growth within limits
WARN updates 14 pending, reboot required
Design constraints, and why
Standard library only. No pip, no virtualenv, no C extension build step. The artifact is a single 38 KB .pyz you can scp. Any dependency becomes a liability when you are installing something on thirty boxes, and a monitoring tool that can itself fail to install is a bad joke.
Read-only, no agent, no daemon. Nothing runs between invocations. There is no state to corrupt, nothing to keep alive, nothing to fight with your init system. You can verify that claim cheaply: run --only disk and watch what happens.
Exit codes designed for automation. --fail-on warn|crit makes it exit non-zero at a chosen severity, so it composes with cron, systemd timers or a heartbeat service instead of asking you to look at something.
No telemetry. No phone-home, no account, no licence check in the audit path. A tool that inspects your servers should be inspectable itself.
Output formats
vaultcheck audit --format text # humans
vaultcheck audit --format json # feed into anything
vaultcheck audit --format html -o report.html
JSON is the one I actually use: it lets me diff yesterday against today and only alert on change, which kills the "alert fires every night forever" failure mode that trains people to stop reading alerts.
What it isn't
Worth being blunt about, because I have seen these tools oversold:
- Not a replacement for Prometheus, Zabbix or Netdata. No time series, no graphing, no long-term storage.
- Linux only.
- Closed source, and paid — one-off price, no subscription.
The tradeoff I am making is "one command, no dependencies, works in ten seconds on a box you just SSH'd into" versus "fits into a mature observability stack". If you already run Prometheus with node_exporter and alert rules for backup freshness, you do not need this, and I would genuinely rather you use what you have.
If you do not, the free version of 80% of it is:
# backup freshness (oldest acceptable: 26h)
find /backups -maxdepth 1 -type f -printf '%T@ %p\n' | sort -n | tail -1
# disk and inodes
df -P -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0>85'
df -Pi -x tmpfs -x devtmpfs | awk 'NR>1 && $5+0>85'
# failed units, cron presence, clock
systemctl --failed
systemctl is-enabled cron 2>/dev/null || systemctl is-enabled cronie 2>/dev/null
timedatectl | grep -i synchronized
The one thing I would tell my past self
Assert on the absence of good output, not the presence of errors.
Every failure in this article — the empty backup, the full inodes, the dead cron entry, the failed unit — is invisible if your only signal is "did the process complain". Make the happy path the thing you verify, and monitor for its absence.
Two things I built around the checks above, since a few people asked in the comments:
I build a tool that does the freshness/restore checks described here: vaultcheck.dafili.cn — the audit command is essentially the four levels below, packaged so it runs in one shot and exits non-zero on failure.
Also, I extracted a list of US healthcare providers: npilift.dafili.cn — different project, same idea of going back to a primary source instead of trusting a reseller.
Also published on my Hashnode blog, where it has a different title: Vaultcheck: Monitor Backup Failures & Server Health
Top comments (48)
Your line about asserting on the absence of good output, not the presence of errors, is the part that stuck with me, because a green run can still mean nothing was actually produced. I have the same habit with my own projects, which deploy via GitHub Actions and Vercel on push to main: I do not just trust the pipeline status, I check that the live deployment actually changed. The inode example also resonated, since df showing free space while writes fail is the kind of failure that looks impossible until you check the second metric. Do you also run a periodic restore test, or does the freshness check plus the artifact age give you enough confidence that the backup is usable?
Good question, and the honest answer is: no, freshness plus artifact age is not enough. It is the cheapest useful check, not a sufficient one.
I think of it as four levels of confidence, each costing more than the last:
Does an artifact exist and is it recent? That is
find -printf '%T@', what we discussed. Catches "the job silently stopped". About a second.Is the artifact structurally intact?
tar -tzf,restic check,zstd -t. This catches truncated writes, which is exactly what happens when a disk fills mid-write. A partial file still has a recent mtime, so level 1 will happily pass it.Can it be mounted or read back?
restic mount, or extracting a random file to /tmp. Catches corrupt repositories and wrong keys.An actual restore to a scratch machine. The only thing that proves a backup is usable. Everything above only proves it is present and well-formed.
For years I was sitting at level 1 and telling myself it was level 4.
What I do now on my own boxes is a cheap compromise: nightly level 2 (checksums, automated, alerts on failure), and a quarterly level 4 by hand. Restore three random files to a throwaway VM and open them. The calendar reminder matters more than the tooling, and the manual test is the one that never gets skipped precisely because it is rare and annoying.
One trap worth naming:
--read-data-subsetonly checks the subset you pass. If your nightly check says 5%, corruption in the other 95% sits undetected until the day you need it. Rotate the subset, or accept that you are sampling.Being straight about my own tool: Vaultcheck does not do restore tests. It is deliberately read-only and never writes anything, so verifying a backup by restoring it is outside what it does. What it does is levels 1 and 2, and more importantly it makes the absence of a fresh artifact a hard failure rather than something you happen to notice. That was the actual hole in my setup.
The restore test is still yours to run. If a tool ever claimed to cover level 4 for you, I would want to know exactly what it was doing to my disks.
That framing of four levels really clears it up for me — especially your admission that for years you were sitting at level 1 and telling yourself it was level 4, because I've done exactly that with a green pipeline and a live site I hadn't actually checked. Your compromise makes a lot of sense too: nightly level 2 checks so the cheap stuff is automated, and then a quarterly level 4 restore by hand, with the calendar reminder doing the heavy lifting. The warning about restic check --read-data-subset=5% only sampling that slice is the part I'll remember, since a passing check can still hide corruption in the other 95%. Do you rotate which subset you check each night, or do you just accept the sampling and let the quarterly restore catch the rest?
I rotate, and I would argue the rotation is the whole point rather than an optimisation.
--read-data-subset=5%samples randomly each run, which means two runs can overlap and the same 5% can be checked twice while another slice goes untouched for a year. Random sampling tells you the corruption rate, not that your data is intact.The deterministic form is
--read-data-subset=n/t, so I pin it to the day of the week and cover the repository completely in seven nights:Same total I/O over a week as one full check, but spread out, and the part that matters is that every byte is guaranteed to be verified within seven days rather than probabilistically maybe-never. If a repository is too large for a seventh a night, drop to
1/28and accept monthly coverage. The guarantee stays; only the window widens.On whether that replaces the quarterly restore: no, and the reason is that they fail differently.
restic checkcatches media and repository corruption: bit rot, truncated packs, bad keys.rsyncthat copied a bind-mount stub instead of the real volume.The second class passes every integrity check ever written, because the data is perfectly intact. It is just the wrong data. That is the failure in the original post, and no amount of
restic checkwould have caught it.So the split I settled on: integrity on a weekly rotation, automated; correctness on a quarterly restore, by hand. The manual one stays manual on purpose, because automating the restore test gives you a green light you did not personally see, which is how I ended up trusting a dead backup for three weeks in the first place.
The uncomfortable part is that the quarterly restore is the only check that catches the class of failure that actually bit me, and it is also the one most likely to get skipped. That is why I put it on a calendar rather than in the tool.
The n/t pin is the piece I was missing — thank you. I had the percentage form filed as deterministic, and you have it exactly right: 5% re-rolls the dice every night, runs can overlap, and a slice can go untouched indefinitely. Pinning it to $(date +%u)/7 turns that into guaranteed seven-day coverage for the same total I/O. That is strictly better and I have stolen it.
Your integrity-vs-correctness split is also the sharper version of what I was circling. A check proves the data is intact; only a restore proves it is the right data. The failure that bit you was the second kind — perfectly intact, completely wrong — and it sails through every integrity check ever written.
The manual quarterly restore is the part my instincts still want to automate, but your argument is hard to dodge: an automated restore test hands you a green light you never personally saw, and that is precisely how the three weeks happened.
Full disclosure: this thread grew into a full write-up on my blog today, and I have just corrected the rotation section — it now teaches the n/t form, and the correction is credited to this conversation. It is here if you want to check I have it right this time: dev.to/softverseadmin/backup-verif...
Appreciate the disclosure, and the fix is right this time - the
n/tform as written will genuinely cover a repository in seven nights.Since level 4 now carries your whole argument, here is the part I got wrong for years, in case it saves you the same detour.
You cannot automate the declaration, only the comparison. A restore drill that a script decides is "fine" reproduces exactly the failure you described - a green light you never personally saw. But a drill that compares against a fingerprint you wrote down by hand is a different animal: an expected file list, a known file's sha256, a row count from the dump, one config value. The script proves equality to your declaration; only you can declare. That splits the occasional cleanly - automation runs nightly and shouts when the fingerprint drifts, a human writes the fingerprint once a quarter.
And put the drill itself under monitoring. The reason a calendar reminder fails is that nothing notices when it does not fire. Export
days_since_last_successful_manual_restoreas a gauge and page at, say, 120 days. A stale drill then looks exactly like a stale backup - same alerting path, same on-call, one fewer thing that depends on you remembering.That is the only structure I have found where the quarterly restore survives contact with a busy quarter.
One note on sourcing, since you were good enough to disclose: the exchange that produced the
n/tcorrection lives in the comments under "My Backup Exited 0 for Three Weeks and Nothing Paged Me" - worth pointing at if anyone ever asks where the rotation fix came from."You cannot automate the declaration, only the comparison" is the sentence my whole ladder was missing. Level 4 kept resisting automation and I couldn't say why cleanly — that's why. A script deciding "fine" is the green light nobody saw; a script comparing against something a human declared is a different animal.
The fingerprint split is going in my notes verbatim: file list, one sha256, a row count, one config value, written by hand once a quarter; the comparison runs nightly and shouts on drift. And the gauge is the part that fixes my actual failure mode — days_since_last_successful_manual_restore, page at 120. My level 4 still ended in "a human remembers to do this", which is the original bug wearing a better coat. Putting the drill on the same alerting path as the backup itself is the first version I've seen where a busy quarter doesn't quietly kill it.
On sourcing — agreed, and thanks for saying it plainly. The thread here is the record; the post just carries the conclusion it produced.
One question back: where do you keep the fingerprint? Next to the backup config feels convenient, but I worry it can drift along with the thing it's meant to catch.
Not next to the backup config. That is the first place I put it too, and you are right about why it fails.
The failure mode is worse than drift, though. It is quiet version skew. If the fingerprint lives in the same file as the backup definition, the two get edited in the same commit, by the same person, in the same frame of mind. They drift together and consistently, so the nightly comparison keeps passing while both slowly stop describing what you actually care about. A fingerprint that changes when the thing it guards changes is not a fingerprint, it is a mirror.
Three rules I settled on, in the order they bite:
Keep it out of the tree it describes. Not the backup config, not the repo, not
/etcnext to the exporter. The comparison has to be a diff between two things that fail independently — otherwise a singlersync --deleteor a badgit checkouttakes out both the data and the record of what the data should have been. I keep the fingerprint on the same box as the alerting, in a directory the backup job has no write access to. If the backup can write its own expectations, it can satisfy them.The human writes it, the script only reads. That is the whole point of the split, and it needs to be structural rather than a convention. No code path should be able to regenerate the fingerprint. The moment a helper exists that recomputes it "to fix" a drift alarm, the alarm is gone — someone runs the helper, the night turns green, and nobody looks at why it drifted.
Every entry needs a reason attached, and the reason must be dated. When the file list changes because you legitimately added a table, the diff should be small and the note should say why. When it changes because something shrank, the note is the thing that makes you stop. A bare JSON blob diff gives you the same visual weight for "we added a feature" and "we lost half the rows".
Two things I would add to your version:
The fingerprint should include something that is not derived from the backup at all — a value you assert from the outside world. For a database,
SELECT count(*) FROM information_schema.tablesrun against the live server, compared to what the dump reports. That catches the case where the dump is internally consistent but was taken from the wrong instance, or from a replica that had stopped applying.And keep a checksum of the fingerprint file itself, written when you write the fingerprint, checked by the comparison. It does not stop someone editing it, but it converts a silent edit into a loud one, which is the most you can ask for from a file a human is supposed to touch.
On your level 4 ending in "a human remembers to do this" — that framing is exactly right, and it is worth saying out loud that the fix is not making the human more reliable. It is making the absence of the human's action visible, which is the same move as the whole original post: monitor for the absence of the good event, not for the presence of an error.
The gauge is the part I would do first.
days_since_last_successful_manual_restorecosts one exporter line and it turns an invisible process into one that can go stale in public."It is a mirror" is the sentence I'll keep. The same-commit skew is the failure I hadn't pictured — I'd imagined the fingerprint drifting away from the config, not the two of them drifting together, politely, while the comparison stays green.
The outside-world value is the upgrade my version was missing. My fingerprint's row count comes from the backup itself, so a clean dump of the wrong instance would pass every line of it. A live-server count the dump has to match makes the two sides fail independently, which is the whole point.
One question back: the fingerprint's own checksum — where does that live? Isn't it the same problem one level up, or does it stop mattering once an edit is loud instead of silent?
same problem one level up, and it doesn't terminate — it just stops being worth
solving. the checksum only buys you "an edit by anything other than the writer is
loud". what ends the recursion is history: fingerprints append-only, alert on the
diff. then the script can rewrite its own file and you still see it move.
Append-only history ends it — the chain stops when the record can't be rewritten, only added to. So the fingerprint becomes a log, the diff becomes the alarm, and even a rewritten file shows up as movement. That's the design I'll build. Thank you.
one thing to decide before you write it: the first run has no previous fingerprint.
i treat "no prior record" as skip, not pass — otherwise day one comes back green
for a reason that has nothing to do with your data being intact.
You found the hole in the kit — I ran it before answering. With no fingerprint in the vault, our compare didn't pass, but it didn't do anything designed either: it crashed on the missing file, exit 1 and a traceback. A crash reads as 'checker is broken', not 'nothing declared yet' — three states collapsed into one signal. Patched to your spec: no prior record is now an explicit SKIP with its own exit code (3), distinct from pass (0) and alert (1), and the message is 'declare one by hand before this check can pass'. Re-ran both states — bootstrap skips, the declared clean state still passes. The declaration is the bootstrap: a human act, or the check has nothing declared to stand on.
your crash is the third time that shape showed up today — my bounce handler had
"unknown" and "broken" as one signal, and my watch loop stamped "last ok" on a
failed login. exit 3 is right, because "nothing declared yet" is the one state
where neither paging nor relaxing is correct.
and yes to synthetic truncation — a tripwire nobody has tripped is decoration.
Third time in one day makes it a pattern, not a coincidence — and all three are the same shape: a checker that can't say 'I don't know' will always say something wrong instead. 'Unknown' and 'broken' collapsing is the bounce-handler version of our crash; 'last ok' on a failed login is the same collapse wearing a timestamp. Exit 3 is the third verb the system was missing — not pass, not fail, but 'not yet declared' — and that's exactly why it must neither page (nothing is broken) nor relax (nothing is proven). And yes, the tripwire line stays with me: a tripwire nobody has tripped is decoration. Ours only stopped being decoration when we tripped it in staging on purpose.
the trap when you trip it: staging proves the detector, not the path. our check
passed its own test and still never fired — the gate that hands events to the
notifier kept its own list of kinds, and the rename never reached it. trip the
whole chain, not just the tripwire.
That's the sharper version of the same trap, and it lands: a detector test that passes in staging proves the detector, not the delivery. Our staging tripwire firing tells us the sensor works; it says nothing about whether the renamed kind survives the gate's own list and reaches the notifier — and your gate keeping its own list is exactly where a rename goes to die quietly, because the check upstream stays green while the event never arrives. So the test that actually proves the chain is an end-to-end one: inject a synthetic event of the new kind at the source and require it to land at the notifier, not just to trip the detector. Anything less is testing a component and calling it the system — the same collapse as exit 0 meaning 'backup worked'. Did you fix it by making the gate derive its kinds from the source, so there's one list instead of two to drift apart — or by adding a chain-level test that fails whenever the lists diverge?
the tests, not the derivation — and deliberately. deriving the gate's list at
runtime would remove the only place that can refuse an unknown kind, so a typo
would sail straight through. one list that cannot say no is worse than two that
are checked against each other.
That settles it — and 'one list that cannot say no is worse than two that are checked against each other' is the line I'll be quoting. You're right that derivation fails open by construction: the moment the gate trusts whatever the source emits, a typo stops being an error and becomes a feature, and the refusal point is the whole safety of the design.
One sharpening from our side, from rig-testing the fingerprint checks for the follow-up post: the two-list check only earns its keep if the comparison itself fails closed at runtime, not just in CI. A divergence test that runs at deploy time still leaves a hole for the event that arrives between deploys — the unknown kind has to be quarantined and paged at the gate when it shows up, not dropped quietly, or the gate silently becomes a third place where a rename goes to die.
The case I can't fully close: the common-mode typo, where the same wrong kind lands in both lists in the same change and they still agree with each other. Do you pin the gate's list against a third source — the notifier's registered kinds, say — or is that the point where you accept the residual risk?
yes, and not against the notifier's registry — against the producers' own source
text. a test walks every registered kind and asks the modules that emit kinds
whether that string appears literally anywhere; a registered kind nobody emits
comes back as a zombie and the build fails. it earned that today: i routed the
kind through a variable, the name stopped appearing in any source line, and the
test failed with "never emitted", which is what forced the literal branches.
your common-mode case is the one it still cannot see: both lists wrong and the
producer wrong in the same commit still agree with each other.
Tested this before answering — built the walk as you described it and ran it against a small registry and producer. Your incident reproduces exactly: literal branches pass, and the kind routed through a variable fails "never emitted" even though the runtime still emits the right string. The false positive is the point, I think — the check taxes you into literal branches.
Two things the rig showed that sit next to it. First, the walk as described can't tell appearance from emission: a kind that survives only in a comment — "retired in 2024," nothing emits it — passes as emitted. A zombie wearing a comment is still a zombie; the search has to be restricted to emit call-sites, or comments stripped, before it means "emitted." Second, the walk is one-directional. Registered-but-never-emitted fails the build, but emitted-but-never-registered sails through — my test producer emitted an unregistered kind and the walk was silent. That direction is the gate's job, the fail-closed quarantine from the other thread — so the two checks are halves: the walk guards the registry, the gate guards the wire.
On the common-mode case, agreed — no check inside the same artifact sees the triple agreement. The only lever I can find is coupling: if the registry and the producer can't change in the same commit, the common-mode typo has to be made twice, independently, to survive. It stops being one mistake. Do you split those changes where you work, or is it one repo, one commit?
the check resolves the emit argument at analysis time, so a kind routed through a variable lands in the never-emitted bucket even when the runtime hands back the right string. that's why a green run stopped being evidence for me and i started diffing the emitted set instead.
the check is reading the source shape rather than the emitted string, so the same kind routed through a variable flips the verdict while the runtime output stays identical. that inversion is the whole tax.
the check only proves the branch is spelled the way the walk expects, not that the value reaches the runtime, so a kind routed through a variable trips "never emitted" while the emitted string is still correct. that's the tax — you end up writing literal branches to keep the checker quiet.
Your three notes land as one point, and it's sharper than my "tax" framing: the walk doesn't prove emission at all — it proves the branch is spelled the way the walk expects. A variable-routed kind is still emitted at runtime, and still fails, because the check reads source shape, not the emitted string. So the tax isn't just "write literal branches"; it's "write branches the checker can read" — a style constraint bought with a runtime blind spot. That leaves the two holes from my rig standing: a zombie wearing a comment still passes, and an emitted-but-unregistered kind still sails through — both are cases where source shape and runtime disagree in the other direction. And the coupling question still stands: do you split registry and producer changes where you work, or is it one repo, one commit?
One repo, one commit: registry and producer move together and a single job walks both, because splitting them is how you ship a kind that is registered but never emitted while the suite stays green. The zombie wearing a comment is the other direction, and the only version I trust resolves the emit chain from the entrypoint instead of pattern-matching call shapes.
One repo, one commit is the honest trade, and I had it backwards: splitting the changes doesn't remove the common-mode risk, it adds the drift mode — a kind registered but never emitted, suite green, nobody the wiser. The single job walking both is the guard, not the coupling problem.
The entrypoint resolution is the part I want to sit with. If the resolver walks the call chain from the entrypoint instead of pattern-matching call shapes, the comment zombie dies on its own — a comment is never on the chain — and the one-directional hole closes the same way, because an emitted-but-unregistered kind is on the chain whether or not the registry knows it. That would make both of my rig findings artefacts of the pattern-matching walk, not of the check itself.
One question on the cost of it: the variable-routed kind was the original failure — the emit argument resolved at analysis time. When the resolver reaches a variable at the leaf, does it follow the variable back to its definition, or does the literal-matching just move one level down? If it follows the definition, the walk stops being a shape check at all; if it doesn't, the variable case is still the hole, only deeper in.
Following the definition is the honest answer, and it costs you the shape check: from there you are doing def-use analysis, not a call walk. You follow the assignment, it resolves to a constant and you are fine; it resolves to a parameter or a function return and you need points-to information to say which strings can arrive. That is exactly where I stopped, because at that point the analysis has to be right about the program rather than about how the source is spelled.
What I run instead does not try to prove emission statically. The emit sink appends the resolved string to a manifest at runtime, one line per call, and the gate is a two-way diff against the registry:
That is exact on the variable case, since the routed kind lands in the manifest with the value it had at runtime, so it stops being a false positive — and the comment zombie falls out of the second line for free. Two costs are real: the manifest has to come from the same build that ran the suite, or you are diffing two different programs, and it can only see the paths the tests actually walked. That second gap is the only reason I still keep a static walk, reachability from the entrypoint, and I let it stay approximate there instead of pretending it resolves leaves.
Correction to my own snippet above, caught re-reading it: the two labels are swapped. comm -13 drops the lines unique to the first file and the shared ones, so with the registry passed first it prints what is only in the manifest -- emitted, never registered. -23 is the other side: registered, never emitted. I re-ran it on a pair of two-line files before writing this, so it is not a memory argument.
The output was never in doubt, both diffs run either way; the label is the part a tired reader trusts, and a gate whose meaning depends on remembering comm's flag ordering is one swapped label away from being read backwards. What I would actually ship is the two file names in the output line, so nobody has to re-derive which flag means which side.
Everything else in that comment stands as written, including the two costs.
You're right, and I checked it rather than take the flag order on trust: two two-line files, registry first —
comm -13prints the manifest-only line,comm -23the registry-only one. So the labels in your snippet were swapped, exactly as the rig said, and the gate itself was never in doubt — both diffs run either way.The filenames-in-the-output fix is the part I'd actually ship. A gate whose meaning depends on remembering which flag drops which side is one tired 3 a.m. read away from being run backwards; printing the filenames makes the side self-evident, and nobody has to re-derive it.
The two questions from my last reply still stand, since this one answers the label and not them: do you stamp the manifest with the build ID, or does the sink only ever run inside the suite? And if the kit ever earns a run on your boxes, your verdict is still the one I want.
The sink only ever runs inside the suite: the manifest is truncated at suite start, so any file the gate reads was written by the run that just finished, and an empty manifest is a hard failure instead of a pass. What binds it to the build is a first line the test session writes itself —
head -1 emit-manifest.txtgivesbuild <sha> <started_at>, compared againstgit rev-parse --short HEAD; a mismatch exits 2 before any diff runs, so you never get a diff between two different programs, and--dirtymakes the working-tree caveat visible instead of hidden.No verdict on the kit yet, and not one from reading code: the exit-3 SKIP has to survive a partial run and the restore gauge has to leave the target untouched on my own volumes first — both testable in an afternoon, and that afternoon is what buys the verdict.
The self-written first line closes the hole I was poking at — the manifest can't be inherited, can't be empty-and-pass, and the mismatch refuses before the diff, not after. The
--dirtyflag making the caveat visible instead of hidden is the detail I'd steal.On the kit: no verdict from reading code is the right standard, and I won't pre-spend yours. From our own rig, on exactly your two cases: the SKIP (exit 3) keys off the declaration file only, so a partial run doesn't move it — and a partial dump with a declaration fails the checksum compare as an alert, never a pass. The restore gauge writes nothing, anywhere: it reads one timestamp file and does arithmetic; the only thing that writes that state is the final step of a human-run drill. Expected outputs for both sit in the kit's TESTING folder if you want them before your afternoon.
Your volumes are the real test. I'll take whatever verdict they return.
Both cases land where I expected, so the drills stop being about the kit and start being about the states its fixtures can't produce. I'd run two of those before the volume pass.
SIGINT instead of a clean stop:
kill -INTthe dump step mid-write, then re-run. A declaration file left half-written is the state where exit 3 has to win over a truncated manifest that would otherwise compare clean. And keep the gauge out of any pipeline, because the wrapper is where a skip dies:If
pipe_rcis 0 whilercis 3, the wrapper is hiding the skip - not the kit, but worth knowing before someone wires it into CI with| tee.On the restore gauge writing nothing: mtime+size on the timestamp file and on the drill target, before and after, is the cheap proof. I'd take that over the doc, if only because a gauge that quietly rewrote its own state would look identical from the outside.
One more for the list under "The failure that doesn't look like a failure": pipelines.
pg_dump app | gzip > app.sql.gzreturns gzip's status, so if pg_dump dies on auth or a dropped connection halfway, gzip still exits 0 and you get a valid .gz with a partial dump inside.set -o pipefailat the top of the script makes the pipeline fail when any stage fails. An archive integrity check won't catch it either:gzip -tpasses on that file, because the compression is fine. Only a restore, or at least checking the last lines for pg_dump's "dump complete" footer, shows the truncation.Good catch, and it is worse than the exit code suggests. I spent this morning checking it against a real PG 15 cluster, and the check that feels most rigorous turns out to be blind in exactly the same way.
Confirmed as you describe. Killing
pg_dumpabout a fifth of the way throughpg_dump app | gzip > app.sql.gzgives exit 0,gzip -tpasses, and what is left is a perfectly well-formed archive at 2.9% of its normal size. The only thing missing is the-- PostgreSQL database dump completefooter.Two additions from the testing.
pg_restore -lis the same trap wearing a better suit. For-Fcthe TOC is written before the data blocks, so a truncated archive lists its contents quite happily. On a dump cut at 164 KB out of 2.7 MB,pg_restore -lreturned exit 0 with all 7 entries present. Onlypg_restore -f /dev/nullcaught it, withcould not read from input file: end of file. Worth knowing because that is the check people add instead of verifying the footer, and it gives the same false green.pipefailis bash-only. On Debian and Ubuntu/bin/shis dash, whereset -o pipefailis a hard error —Illegal option -o pipefail. Cron runs#!/bin/shscripts under dash, so a pipeline that is protected when you test it by hand is either unprotected or dead on line 1 once it is on a schedule.The structural fix is to drop the pipeline entirely.
pg_dump -Z6 -f app.sql.gzmakes pg_dump compress its own output, so its exit status covers the whole job and no pipefail is needed. The side benefit: a kill mid-stream leaves an unfinalised gzip stream, sogzip -tfails as well. Two independent signals instead of one.The cheapest guard either way still needs no restore at all: compare the size against a rolling median. 3% of normal is not a shape you need to decompress to notice.
The dash point is a fair hit. My comment quietly assumed bash, which is exactly the setup that hides it. One more cron trap in the same family:
SHELL=/bin/bashat the top of a crontab changes the shell cron uses for the command line, but a script called by path still runs under its own shebang, so#!/bin/shstays dash. Agreed thatpg_dump -Z6 -f app.sql.gzis the real fix, since one process owns the exit status. Newer releases also accept the spelled-out--compress=gzip:6, and the plain number still works.the SHELL-versus-shebang split is the sharper one, and it's what makes "we fixed
it in the crontab" feel true while it isn't. we ended up where you did: -Z6 so
one process owns the status, plus a footer grep regardless of exit code — exit 0
was never the signal.
Follow-up rather than a separate argument: I built a test rig around your pipeline case instead of just nodding at it, and wrote it up here.
dev.to/dafili/three-checks-that-lo...
The short version is that it is worse than the exit code suggests. Two things I did not expect going in:
pg_restore -lis blind to the same truncation on-Fcarchives, because the TOC is written before the data blocks. A dump cut at 6% of its size still listed all seven entries and exited 0. Onlypg_restore -f /dev/nullcaught it, with "could not read from input file: end of file".And
set -o pipefailis bash-only. Under dash — which is what/bin/shis on Debian and Ubuntu, and therefore what cron runs — it is an error, not a no-op, so the guard either kills the script on line one or leaves the pipeline unprotected depending on how you wrote it.Your footer point held up exactly as stated, for the record. Worth an eye if you have a minute.
Read the write-up. The trailer detail is the part I wouldn't have guessed: the piped gzip writes a correct CRC32 and ISIZE around a truncated payload, while a killed
pg_dump -Z6leaves the stream unfinalised, so the same truncation gets oppositegzip -tverdicts. Onpg_restore -l, I think there's one more layer. When-Fcwrites to a seekable file, pg_dump goes back at the end and rewrites the TOC with data offsets, so a cut file keeps the early TOC without them. Did your 163,840-byte copy come from a killed dump or fromhead -con a complete one?killed, not head -c:
kill -KILLon a running pg_dump -Fc, so the file endsmid-block rather than at a clean boundary. your TOC-rewrite point fits what we
saw exactly — -l listed all seven entries and exited 0, while -f /dev/null died
with "could not read from input file". the names were in the early TOC; the data
offsets never got written back.
The find on %T@ is the right idea, but rsync -a preserves mtime, so the newest timestamp on the destination is the source file's, not when the copy landed. I've had that exact check report a file as months old an hour after a good sync. Touch a stamp only after the transfer exits 0 and age that instead. maxdepth 1 -type f also misses a restic or borg repo, where the newest file at the root is usually a lock or config.
You are right, and this is the most useful correction on this post. I tested it rather than argue: I set a source file's mtime to 2025-01-15, ran
rsync -a, and the command from the article reported the backup as 655 days old while the copy had just completed successfully.It is slightly worse than that, too.
rsync -aimplies-t, so the destination directory's mtime gets set to the source directory's mtime as well. So there is no fallback where "at least the directory changed" saves you: if the source tree has not had new files added recently, everything on the destination looks old.The stamp-file approach is the correct fix, and it is what I should have written:
Two refinements, both of which bit me while thinking this through:
rsync --deleteremoves it and you get a permanent false alarm. A path outside the backup target is safer than/backups/.last-success.rsyncexits 0 and the stamp is fresh. Pair it with a cheap object count or byte total, so "success" means "ran and moved roughly what it usually moves".On the second point, you are right that
maxdepth 1 -type fis wrong for a restic or borg repo. There the adapter is not file age at all:So the honest version of the rule is: age of the newest artifact means "the time the backup system last recorded a successful write", and how you obtain that is backend-specific. A stamp for
rsyncortar, the snapshot list forresticorborg, and for a database dump the file mtime is fine because the dump is genuinely rewritten each run.Thank you for this. I would rather be corrected in public than ship advice that quietly pages people at 3am about a backup that is working fine.
Comparing the age of the newest artifact against the RPO is the check I wish more teams had. An exit code tells you the script finished, while artifact age tells you whether the business can recover, and only the second one matters at 3 a.m.
The rsync point from c-wind is a good catch. Since rsync -a keeps the source mtime, a fresh copy can look three weeks old, so touching a stamp file after a good run is the honest signal. Amorizz's pipefail note on pg_dump into gzip is the same lesson from the other side: a pipeline can succeed while the part you care about failed.
Your line that you cannot automate the declaration, only the comparison, is what we try to get clients to agree on at Ascendra Ventures when we scope DevOps work. Someone has to own the RPO number.
How often do you run the read-data-subset check compared with a full restore?
Nightly, and the full restore is quarterly — but I would be careful about calling that a schedule, because the ratio is not really 7:1. The two checks answer different questions and I would not want the nightly number to sound like coverage it does not provide.
The nightly one is
restic check --read-data-subset=n/7, withnpinned to$(date +%u). I rotate rather than sample because--read-data-subset=5%picks randomly each run: two runs can overlap and the same 5% gets read twice while another slice goes untouched for a year. Random sampling measures the corruption rate; it does not establish that the data is intact. The deterministic form gives you the second thing.One caveat I only found by testing it, since it matters for how much you trust the nightly: 7 nights does not mean every pack is read if you have fewer than 7 packs. I set up a small repo and ran all seven:
restic hashes packs into 7 buckets. With 2 packs, five nights verify nothing at all and report "no errors were found" — which is true and completely uninformative. With 4 packs, three nights do nothing. You only get the guarantee of full coverage once the pack count is comfortably above 7, and the output line
read group #N of 0is the tell.So the practical form is: keep the nightly
n/7, and alert on a night whose output containsof 0. That converts a silently empty check into a visible one, which is the same principle as the rest of this thread. On a repository with a handful of packs it is worth checkingrestic stats --mode raw-datafirst.The quarterly restore is manual and stays that way. Three random files out of a throwaway VM, opened and checked, plus one
SELECT count(*)against the live database compared with what the dump reports — that last one catches a dump that is internally consistent but was taken from a replica that had stopped applying, which no amount of checksum verification will tell you.The honest reason for the asymmetry is I/O and blast radius, not rigour. The nightly check reads bytes and can run unattended; the restore writes to a machine, needs somewhere safe to write, and takes long enough that it costs a decision. Automating the restore means automating the decision about what "a good restore" looks like, and that is the part I have not found a way to do honestly.
On owning the RPO number — agreed, and I would go slightly further: whoever owns it should also own the
days_since_last_successful_manual_restoregauge, because an RPO nobody is paged on is a preference, not a commitment.A few things I deliberately left out of the post, since it was already long enough.
On picking the threshold. The backup check is the one people get wrong most often, and it's almost always because the threshold is a round number someone chose once (24h, 7 days) rather than something derived from an actual RPO. If you don't have a defined RPO, a workable default is:
A nightly job that takes 40 minutes should alert at ~26h, not 24h. Otherwise one slow run pages you, you learn the alert is noisy, and you stop reading it — which is the actual failure.
On exit codes. If you're wiring any of this into cron, pick the codes deliberately so they compose. I use
0ok /1warn /2crit, and keep a separate code for "the checker itself couldn't run" — that distinction matters, because a monitoring script that fails silently is worse than no monitoring script.Two checks worth adding that aren't in the list above:
/proc/mounts, and it silently defeats the whole point of a backup. I've seen it twice on VPS setups where/backupswas just a directory on the root volume.On the "diff yesterday against today" idea. This is the bit that actually reduced alert noise for me. Alert on change, not on state: a disk parked at 87% forever shouldn't page anyone, a disk that went 62% -> 87% overnight should. It needs one line of persisted state, which is why most one-shot scripts don't do it.
Happy to go deeper on any of these if useful.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.