I'm thinking more from the point where your tax issue isn't resolved and you have no recourse at all, because the AI has final say.
Your cancer is undiagnosed because there is an issue with the AI. You can't get a second opinion, so just die in pain in your house and literally can never speak to a real medical professional. Or the AI can be automatically tuned to dismiss patients more readily as hospitals are getting a bit busy. I doubt it would have any moral objection to that.
If your tax issue isn't resolved and the AI has the final say, the problem is that the AI is the final authority, not that the AI isn't good for the (presumably vast majority of) people that it can help.
Same with the cancer diagnosis:
Both of these arguments are along the lines of the "seatbelts are bad because in 0.2% of accidents people get trapped in cars because of them."
This AI will dramatically improve outcomes for an overwhelming majority of people. Sure, we'll all think it sucks, just like we think phone queues suck now -- even though they are vastly superior to the previous system of sending paperwork back and forth, or scheduling a phone meeting for next Tuesday.
Your cancer is undiagnosed because there is an issue with the AI. You can't get a second opinion, so just die in pain in your house and literally can never speak to a real medical professional. Or the AI can be automatically tuned to dismiss patients more readily as hospitals are getting a bit busy. I doubt it would have any moral objection to that.