"Hm, maybe I should double check ChatGPTs output... Hey, ChatGPT, does your output make sense?" - "Yeah, my output definitely makes sense". "Are you sure?" - "Yeah".
People are scarily willing to take eloquent text on trust if it says something that benefits them and sounds plausible.
The biggest problem with ChatGPT is that it sounds like a smart adult that people want to give the benefit of the doubt because it agrees with them a lot instead of like a child that people would be more wary of trusting blindly.
This willingness most people show to jump on the bandwagon when hearing confident sounding language actually explains a lot of societal problems. Also a lot of workplace dynamics, and the career of a lot of very powerful and rich people.
https://news.ycombinator.com/item?id=36092914
Particularly relevant is the affidavit in which the lawyer tries to explain to the Court what happened:
https://storage.courtlistener.com/recap/gov.uscourts.nysd.57...