▲
11
How do I stop one broken app from eating all my log storage?
An app started logging the same stack trace ten times a second over a weekend. By Monday it had taken a serious bite out of our quota.
An app started logging the same stack trace ten times a second over a weekend. By Monday it had taken a serious bite out of our quota.
Three layers and you want all three.
At the source, rate limit. rsyslog has
$SystemLogRateLimitInterval, most agents can cap per file.In the pipeline, deduplicate. Identical messages in the same minute collapse to one with a count. That is the one that saves you in exactly your situation.
At the platform, alert on volume rather than only on content. A source that triples its rate is worth knowing about whether or not the messages are interesting. Almost nobody does this and it is the cheapest of the three.