▲
19
How do I alert on something not happening?
We want to know when the nightly backup stops logging its success message. Every alerting system I have used only triggers on events that exist.
We want to know when the nightly backup stops logging its success message. Every alerting system I have used only triggers on events that exist.
Hardest alert to build, most valuable one to have.
The shape is a scheduled check that asks a question rather than a rule that reacts to an event. Did we see at least one matching event in the last N hours, if not, alert.
The trap is that it fires on the first bank holiday and everyone learns to ignore it. Give the window generous margin, and monitor the check itself, because a silence detector that silently dies is worse than not having one.
Same pattern covers a host that stopped sending logs entirely, which is the version that actually saves you.