Alert on staleness, not on health
For eleven days this summer, my secondary DNS server answered every query, passed every health check, and enforced a configuration that was quietly going out of date. Nothing was down. That was the problem.
My Park network runs two AdGuard Home instances: a primary on a virtual machine, a replica on my NAS. A small sync container copies the primary's configuration (blocklists, custom rules, client settings) to the replica every thirty minutes.
On 17 July I changed one port binding on the primary so that its admin interface would be reachable from the LAN instead of only through an SSH tunnel. Docker replaces a port binding; it does not add a second one. The address the sync container relied on, the VM's own loopback, stopped listening. Every thirty minutes the sync tried, received "connection refused", wrote one line to its log, and waited for the next run.
The sync container stayed up. Both DNS servers stayed up. Both answered queries. Everything I watched measured liveness, and everything was alive.
On 27 July, a log analyzer that reads container errors flagged the repeated failures. Once the address was fixed, the first successful sync took 9.1 seconds and pushed eleven days of drift to the replica: a new blocklist, nine user rules, a blocked-services schedule and a client definition. For eleven days, half of my DNS had been applying an older policy than the one I believed was in place.
Health was the wrong question. A health check asks: do you respond? The question that mattered was: when did you last do your job? Those are two different signals, and only the first one comes for free.
Three rules I took from it:
- Alert on the age of the last success, not on the presence of a process. A job meant to run every thirty minutes that has not succeeded for two hours is broken, whatever its container status says.
- Let the job record its own success. The job writes when it last finished well; the alert compares that time with the expected interval. The monitor should not have to guess.
- Treat silence as a finding. No errors from a job that should run every half hour can simply mean it never ran.
I have started applying this beyond DNS. One example: a check on the age of the last completed Time Machine backup, rather than on whether the backup service is running. A backup that is configured but has not completed recently is not a backup.
None of this is specific to a homelab. Database replicas, threat-intelligence feeds, antivirus signatures, the documents behind a search index: each can be reachable and stale at the same time, and a green dashboard will not tell you which.
In my case, the fix was a single line. Finding it took eleven days, because nothing I measured was designed to notice that time was passing.
What in your infrastructure is up, healthy, and quietly out of date?