Canary deploy dude (or dude-ette), route 0.001% of service traffic and then slowly move it over. Then set error budgets. Then a bad service wont "bring down production".

Thats how we did it at Google (I was part of the core team responsible for ad serving infra - billions of ads to billions of users a day)