> Datadog makes it far too easy to misconfigure things and blow up costs
I'll give you a fun example. It's fresh in my mind because i just got reamed out about it this week.
In our last contract with DataDog, they convinced us to try out the CloudSIEM product, we put in a small $600/mo committment to it to try it out. Well, we never really set it up and it sat on autopilot for many months. We fell under our contract rate for it for almost a year.
Then last month we had some crazy stuff happen and we were spamming logs into DataDog for a variety of reasons. I knew I didn't want to pay for these billions of logs to be indexed, so I made an exclusion filter to keep them out of our log indexes so we didn't have a crazy bill for log indexing.
So our rep emailed me last week and said "Hey just a heads up you have $6,500 in on-demand costs for CloudSIEM, I hope that was expected". No, it was NOT expected. Turns out excluding logs from indexing does not exclude them from CloudSIEM. Fun fact, you will not find any documented way to exclude logs from CloudSIEM ingestion. It is technically possible, but only through their API and it isn't documented. Anyway, I didn't do or know this, so now i had $6,500 of on-demand costs plus $400-500 misc on-demand costs that I had to explain to the CTO.
I should mention my annual review/pay raise is also next week (I report to the CTO), so this will now be fresh in their mind for that experience.
That’s just the sort of hypothetical scenario that kept running through my head as I tried to find a way for us to use Datadog. I even particularly wanted to use the CloudSIEM product. Bummer.
I'll give you a fun example. It's fresh in my mind because i just got reamed out about it this week.
In our last contract with DataDog, they convinced us to try out the CloudSIEM product, we put in a small $600/mo committment to it to try it out. Well, we never really set it up and it sat on autopilot for many months. We fell under our contract rate for it for almost a year.
Then last month we had some crazy stuff happen and we were spamming logs into DataDog for a variety of reasons. I knew I didn't want to pay for these billions of logs to be indexed, so I made an exclusion filter to keep them out of our log indexes so we didn't have a crazy bill for log indexing.
So our rep emailed me last week and said "Hey just a heads up you have $6,500 in on-demand costs for CloudSIEM, I hope that was expected". No, it was NOT expected. Turns out excluding logs from indexing does not exclude them from CloudSIEM. Fun fact, you will not find any documented way to exclude logs from CloudSIEM ingestion. It is technically possible, but only through their API and it isn't documented. Anyway, I didn't do or know this, so now i had $6,500 of on-demand costs plus $400-500 misc on-demand costs that I had to explain to the CTO.
I should mention my annual review/pay raise is also next week (I report to the CTO), so this will now be fresh in their mind for that experience.