Plenty studies show that these models are better at catching and diagnosing than even a board of doctors are. Doctors are good at other things, and I hope the future will allow doctors to use these models together with their practice.
The problem is when the ai makes a catastrophic prediction, and the layman can't see it.
Yes, and I am pretty sure this is already an established phenomenon; AI and ML in general are able to apply domain specific algorithms better than the people who wrote them, because of how much it gets exposed to it during training.
as a layman, I can't see errors that professionals make either. I trust them, and later I experience the consequences of their mistakes. sometimes catastrophically.
The problem is when the ai makes a catastrophic prediction, and the layman can't see it.