The cost thing was just looking at the bill. I've worked at some companies where engineering didn't even have access to the AWS bill, for example, which is rather unfortunate, since engineering is where the knowledge of "wow, this pie slice shouldn't exist" is. (I do think finance could? should? have noticed that bill, since I in part noticed it b/c the monthly total was too high. But I might have been looking at a more broken-out number than the actual, final total there were some other things that might have gotten mixed in that would have dulled the signal enough to make it plausibly missable.
But after seeing the number, it was a pretty quick task to break it out into various areas over time (like total VM$/time, total storage$/time), find the category that grew, and repeat until the offender was found.
The script thing was mostly about time. Bash's output was only so intelligible (it was basically |grep|awk|grep sort of thing) and engineers would get confused, ask questions, and it soaked up enough time to make me say "let's rewrite this so I never have to answer these questions again".
But after seeing the number, it was a pretty quick task to break it out into various areas over time (like total VM$/time, total storage$/time), find the category that grew, and repeat until the offender was found.
The script thing was mostly about time. Bash's output was only so intelligible (it was basically |grep|awk|grep sort of thing) and engineers would get confused, ask questions, and it soaked up enough time to make me say "let's rewrite this so I never have to answer these questions again".