I have worked in finance tech all my career, so I'm not sure how other orgs work, but this is extremely pertinent. Large investment banks react exactly like this.
I spent a miserable year trying to convince people that they were over-reacting to an outage and there was a very simple solution to the exact problem that occurred. But when senior managers see their jobs at risk because of a repeat, they'll mandate that the entire department review their code for similar issues and remediate. They'll also listen to the loudest voices who somehow come up with massively over-engineered solutions.
Another example, we had a password expire which caused an outage on our trading stack. The amount of effort that went into stupidly convoluted hand-crafted solutions ensuring this "didn't happen again" was laughable. And in the end, after more than a year of work, the whole thing was abandoned in favour of a much simpler centralised solution that should have been done from the start.
I spent a miserable year trying to convince people that they were over-reacting to an outage and there was a very simple solution to the exact problem that occurred. But when senior managers see their jobs at risk because of a repeat, they'll mandate that the entire department review their code for similar issues and remediate. They'll also listen to the loudest voices who somehow come up with massively over-engineered solutions.
Another example, we had a password expire which caused an outage on our trading stack. The amount of effort that went into stupidly convoluted hand-crafted solutions ensuring this "didn't happen again" was laughable. And in the end, after more than a year of work, the whole thing was abandoned in favour of a much simpler centralised solution that should have been done from the start.