If they redact all identifying information, it would most likely be legally Kosher. However, there is an extreme abundance of caution in the healthcare industry regarding everything surrounding HIPAA. Merely questioning the legality of something can cost millions of dollars in lawyers' fees. Therefore even miniscule chances of something being legally challenged (e.g. plugging patient information into an LLM) would most likely be deemed too risky. And frankly, hospital administrators will not want to risk their careers over trying out what they perceive to be a glorified chatbot.

Tl;dr: When it comes to HIPAA, risk aversion is the name of the game.

If you redact all identifying information from a patient case file, it will likely become almost useless. Anything that describes a person in any way is potentially personally identifying information.