The real issue here (sorry for true Scotsman-ing) is that they were using an untested combination of configuration and binary release. Configuration and binaries can be rolled out in lockstep, preventing this class of issues.

Of course there were other mistakes here etc., but the issue wouldn't have been possible if this weren't the case.