My recent use case is needing an alert if a certain metric of my system exceeds a threshold or is not present. The simplest viable alerting tool might take these parameters:
* alert threshold
* comparison operator
* expected datapoint interval (in units of time)
* whether to consider missing data breaching (boolean)
* if missing data is breaching, margin of skew allowed (in units of time)
(assume the tool magically has access to metric source, that’s not important)
I’m not sure whether the concept was named prior to this tool [0], but this tool’s name has become a colloquial term: a “Dead Man’s Snitch” style of alert. Once familiar with the idea, I found an exception-monitoring vendor [1] that I was already using at the time had built this feature in. I quickly found use cases and was broadly able to eliminate large volumes of “cronspam” and email filters in favor of the much more elegant construct — because it was so easy to implement and maintain, it was viable for a whole long tail of second class system metrics that otherwise didn’t deserve proper instrumentation.
CloudWatch on the other hand is lower level and more flexible (naturally with higher learning curve). It fundamentally provides similar value by offloading alerting state and logic to a trusted managed system, but is distinctly inferior for my use case:
* It treats missing metrics as literally breaching the threshold and makes no semantic distinction, which spikes the whole abstraction. You can (only partially) workaround this by splitting the alert into two distinct alerts.
* It doesn’t have a “permitted skew in units of time parameter”; you need to do the arithmetic and translate it into their parameters. This spikes the abstraction and makes the alert logic opaque to my team members and future self.
