Health issues aren’t addressed with accurate information, they’re addressed by understanding the needs of the individual. Even if GPT-4 could guarantee accuracy when discussing self-harm, that would not necessarily be the right answer from the perspective of ensuring GPT-4 does the most amount of good.

If a friend told me that they were suicidal, I could explain to then in great detail about the nuances of depression and medication and suicidal ideation and how to effectively harm themselves if that’s what they want, but I know that is probably not the right answer, and the right answer is actually, “I’m here for you and I will help you get professional help”.

Harm reduction often involves helping people do dangerous things more safely (like safe drug injection) but that’s one component of helping people, the key to harm reduction is the long term investment in addressing the problem. Safe injection, for example, is often married with further healthcare. GPT-4 can’t do that and so telling you to go to a healthcare professional instead is going to have a much better outcome.

> Health issues aren’t addressed with accurate information, they’re addressed by understanding the needs of the individual. Even if GPT-4 could guarantee accuracy when discussing self-harm, that would not necessarily be the right answer from the perspective of ensuring GPT-4 does the most amount of good.

That argument could be used for removing most health information from the internet, restricing books on the topic to people with a medical license, etc.

I agree that ideally any chatbot built on top of GPT-4 should do more, like asking further questions, following up in later conversations etc. And as others have pointed out, GPT itself should point out even better methods to satisfy the expressed immediate need (ice cubes instead of cutting). But saying "Sorry dave, I can't do that. Ask someone else." doesn't sound like the right approach.

Because you avoid introducing any additional harms.