You've hit upon one of the most obvious ways to improve the safety of these systems. Deploy the systems more broadly, without giving them active control.
Then, you can start to identify situations where the driver's actions were outside of a predicted acceptable range, and investigate what happened.
Additionally, if you have a large pool of equipped vehicles you can identify every crash (or even more minor events, like hitting potholes or road debris) and see what the self-driving system would have done.
The realistic problem is that Uber doesn't give a shit. As such, deployment will never be optimized for public safety. It will be optimized for Uber's speed to market.
Then, you can start to identify situations where the driver's actions were outside of a predicted acceptable range, and investigate what happened.
Additionally, if you have a large pool of equipped vehicles you can identify every crash (or even more minor events, like hitting potholes or road debris) and see what the self-driving system would have done.
The realistic problem is that Uber doesn't give a shit. As such, deployment will never be optimized for public safety. It will be optimized for Uber's speed to market.