Most people should be very uncomfortable giving a random gig economy worker access to their personal email accounts that act as fallback authentication for everything in their lives.
The fact that our AI systems have this level of trustworthiness is a big problem for harnessing their potential, since you want them to be a lot more trustworthy.
But AI is even worse, it has no sense for when things are weird and it is under attack. If you sent a hundred messages to a human trying slight variations of tricks on them, they would know something was wrong and they were under attack, but an AI would not.
The fact that our AI systems have this level of trustworthiness is a big problem for harnessing their potential, since you want them to be a lot more trustworthy.
But AI is even worse, it has no sense for when things are weird and it is under attack. If you sent a hundred messages to a human trying slight variations of tricks on them, they would know something was wrong and they were under attack, but an AI would not.