Hacker News new | past | comments | ask | show | jobs | submit login

Not limiting access to production.

It’s both an operational and technical decision because it results in inadequate tooling and specials.

A service I know well had an issue, and as a result they started doing what modern services do - nuking things going it will clear as clean versions come up.

But unknown to the ops team, they nuked a bunch of custom stuff that no one knew how to build or really what they contained. Developers with direct access to production had rolled them out.

7 days of partial outage later they covered this up by hiding parts of the app until they could straighten things out.




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: