That is one of the reasons why Brendan Gregg's USE [1] methodology is so great. USE stands for utilisation, saturation, errors. For every component, resource, or subsystem you should have at least a metric for each of these. Utilisation tells you: How much is it used? Saturation tells you: How near is it to the capacity limit or how much does it slow down because of load, and errors tell you e.g. when k8s pods restart all the time.

[1]: https://www.brendangregg.com/usemethod.html