DEV Community

Airat Iunusov
Airat Iunusov

Posted on Edited on AI-assisted

I tested my own docs like a stranger. They failed 24 times.

I wrote a monitoring stack (Prometheus, Grafana, Alertmanager) and a README that says "it just works". It had always worked on my own server. But I had set that one up by hand, over time, from memory. I had never once followed my own README from the first line to the last.

So I took a clean VM and did exactly that, the way a newcomer would. First on Ubuntu 22.04, then on 24.04 and 26.04. It took about a day.

I found 24 problems. Three of them stuck with me, because I'm the one who got them wrong.

The alert that fired on a healthy machine

On Ubuntu 22.04 a fresh VM with nothing running went to pending on a critical alert: FilesystemReadonly. My first thought was that I'd broken something.

It was my rule. systemd 249 mounts a read-only ramfs for its own needs, and my alert counted it as a failing disk. Ubuntu 24.04 and 26.04 don't do this, so on newer systems I'd never have noticed.

The fix is one word: add ramfs to the exclusion list. The more interesting part is that my own checks had missed it. I was verifying that targets are up and dashboards load. I wasn't verifying that a healthy system is quiet. Now /api/v1/alerts must be empty on every fresh stand.

The restore that restores nothing

My backup doc explained how to make a backup. Restore was one sentence without commands. So I tried it.

The Grafana archive has a top-level grafana/ directory, the Prometheus one doesn't. Unpack Grafana to the wrong path and nothing fails. You keep the old data and believe you restored.

I wrote the steps down and ran them. To be sure it really restored something, I created a folder with a marker in Grafana before the backup. After the restore it was there, and all 7 targets were up.

A backup you haven't restored is a hope, not a backup.

"Ansible 2.9+"

The README said Ansible 2.9 or newer. I had never run it on 2.9, I just wrote the number.

I tried. On 2.9 the syntax check passes, then the run fails on the target with ModuleNotFoundError, a known clash with Python 3.12. From 2.12 to the latest 2.21 everything worked, and the second run gave changed=0. The README now says 2.12+.

Everything else

The other 21 are smaller: a compose block in the wrong place, a stale link, flags in an example config that no code reads, Alertmanager needing restart instead of up -d. All fixed in 15 pull requests, #26 to #40.

What I did not check: a real x86_64 machine (I used emulation), Debian and RHEL, load.

A check I ran by hand once is a snapshot. The next version bump will break something again, so the next step is a CI smoke test: compose up, targets up, Ansible twice with changed=0. Docs should run like code.

How do you check that your install docs still work: CI, a manual run before a release, or you find out from an issue?

Repo: github.com/Airat71/monitoring-stack

Disclosure: I used an AI assistant to run the repetitive checks and to help draft this text. The findings, the decisions and the fixes are mine.

Top comments (8)

Collapse
 
devenquan profile image
Zhang Enquan •

"The newest version hid it" is the line I keep relearning.

I ran the same exercise on my own tool and found exactly two problems — because I ran it once, on one machine, in one state. You found 24 because you varied something. That gap isn't diligence, it's sample size.

Same shape showed up in a measurement I published: 10,000 real text samples through the actual tokenizer. Prose came in at 4.58 characters per token and I was ready to write "about 4.5" in the README. Structured data came in at 3.78. One content type I rarely feed it moved the number 18% — meaning the sentence I wanted to write was true for exactly the input I happened to test with.

Worth one more pass to your method: after "follow the README from line one on a clean VM", do it once more on a machine that already has stuff on it. That third run finds a class neither of the first two can.

Collapse
 
airat71 profile image
Airat Iunusov •

Thanks, fair point. I varied the conditions and you ran it once, so the gap is sample size, not diligence. A run on a machine that already has leftovers (busy ports, old volumes, an earlier install) is something I didn't cover. Worth adding to the method.

Collapse
 
devenquan profile image
Zhang Enquan •

Agreed - and that dirty-machine gap maps to exactly what I keep hitting with token estimates: run against clean benchmark prompts and you undercount errors, the same way a fresh machine undercounts install failures. The leftovers are the real test subjects; the docs just have to survive them. If the method gets a part two, that's the run worth publishing.

Thread Thread
 
airat71 profile image
Airat Iunusov •

Part two is up: dev.to/airat71/i-installed-my-stac...

You were right about clean inputs hiding failures. The sharpest case: after a port conflict, Prometheus showed Up (healthy) with no network, because the healthcheck can't fail.

How do you handle it with token estimates: a "dirty" prompt set, or real traffic?

Collapse
 
yash_thokal_a1899d36b0351 profile image
YASH THOKAL •

Respect for actually following your own README on a clean VM. Almost nobody does that, and the ramfs alert is a perfect example of a bug you can only find from the outside: your own server was "correct" because you'd fixed it by hand years ago.

Did you end up automating any of it? Even running just the install steps in CI on a fresh Ubuntu image would catch a lot of the 24 the next time something changes.

Collapse
 
airat71 profile image
Airat Iunusov •

Thanks. Yes: a smoke workflow runs on every pull request and weekly on a fresh Ubuntu runner. It starts the stack with Compose and with the Ansible playbook, checks that all scrape targets are up, then runs the playbook again and requires changed=0. Since v2.3.2 it also checks upgrades from earlier versions. Still manual: the ansible-core range and restore.

Collapse
 
docsalive profile image
DocsAlive Team •

A practical mix is a clean-VM smoke test on each relevant PR and a fuller manual walkthrough before releases. For this stack, the CI job can run the documented setup, wait for readiness with a bounded timeout, query the targets and alerts APIs, then run Ansible a second time and check its recap for changed=0. Preserve logs on failure so a timeout is distinguishable from a bad assertion.

For restore, create a unique marker before backup, remove or replace the data, restore, and verify that marker rather than relying only on a successful command. Test the minimum supported Ansible/Python combination as well as the current one; document the tested combinations rather than inferring a minimum.

Collapse
 
airat71 profile image
Airat Iunusov •

Thanks, this is close to what runs now: a smoke test on every PR and weekly, readiness with a timeout, targets and rules through the APIs, a second Ansible run with changed=0, logs kept on failure. Two items are still manual: restore (checked by hand with a marker) and the ansible-core minimum (2.12 to 2.21 by hand, CI uses the latest). Restore is the gap I am most aware of.