▲
19
How do I alert on something not happening?
We want to know when the nightly backup stops logging its success message. Every alerting system I have used only triggers on events that exist.
We want to know when the nightly backup stops logging its success message. Every alerting system I have used only triggers on events that exist.
Worth knowing this is a PCI requirement now rather than just good practice. 10.7.2 covers failure of the logging mechanism itself and it stopped being future dated in March 2025.
We ended up doing this with a heartbeat instead. Every host logs a line every 15 minutes and the check looks for the heartbeat rather than for real events. Less elegant, far fewer false alarms around holidays.
Hardest alert to build, most valuable one to have.
The shape is a scheduled check that asks a question rather than a rule that reacts to an event. Did we see at least one matching event in the last N hours, if not, alert.
The trap is that it fires on the first bank holiday and everyone learns to ignore it. Give the window generous margin, and monitor the check itself, because a silence detector that silently dies is worse than not having one.
Same pattern covers a host that stopped sending logs entirely, which is the version that actually saves you.