I once really urgently needed `nmap` to do some production debugging ASAP. Unfortunately, the security tools would flag this immediately on my machine, as I knew this from previous experiments. Solution - compile my own binary from sources, then quickly rename it. I assume that this "workaround" was totally fine for sec department. At least production got fixed and money kept flowing.
> At least production got fixed and money kept flowing.
You were denied the tools to get your job done. You've put yourself at risk by applying an unapproved workaround.
Never ever do this (unless you hold substantial shares). Let the company's bottom line take the fall. If that's the only thing they care about, that's your only way to make the problem visible.
Unfortunately the real world isn't black and white. Yes, according to the company policies, I should watch the world burn and do nothing, while looking at the company bleeding money due to customers SLA being broken. Of course, after submitting a ticket to get nmap approved, which takes days. Extra points if I'm on oncall, then racking that sweet incident money is great.
But the underlying SRE culture here is that, if you know what you are doing and have a functioning brain of a responsible person, you'd be forgiven a jump over the fence, if it means putting out a fire on the other side of it. We aren't kids.
There’s a middle ground. Get the appropriate stakeholders involved in the decision, including security. Let security be the ones to keep the system down, if it cones to that. Or, let the business operations folks make the decision to go over security’s head. Either way, this is not something an engineer tasked with fixing an outage should be making the decision on.
Engineers _should_ have leeway in how they resolve issues. As I read, though, you have a company policy which explicitly disallows the action you needed to take to fix the problem (if I misread, my apologies). Getting the stakeholders involved is the responsible thing to do when policies need to be broken.
Ideally, the way this kind of situation gets handled should be documented as part of a break-glass policy, so there’s no ambiguity. If that’s not the case, though, the business should get to decide, alongside the policy maker (e.g.: security), whether that policy should be broken as part of an emergency fix, and how to remediate the policy drift after the crisis.
If you’re all tight enough that you’re allowed to make these kinds of decisions in the heat of the moment, that’s great, but it should be agreed upon, and documented, beforehand.
Well I found out the hard way that company culture or values can mean nothing if you don't CYA. Granted, the shop was small enough that our team was in charge of both the security policies and ops, but still, on one unfortunate occasion I've stepped outside my area of responsibility to "do what's right" and got punished. The next time I've been in a similar situation - well, I've walked away from the fire and grabbed the popcorn.
By the way, I'm still burnt out. This work is stressful. Don't let it take away what's already scarce for you.
I once really urgently needed `nmap` to do some production debugging ASAP. Unfortunately, the security tools would flag this immediately on my machine, as I knew this from previous experiments. Solution - compile my own binary from sources, then quickly rename it. I assume that this "workaround" was totally fine for sec department. At least production got fixed and money kept flowing.