The thing with Buddhism and perhaps any genuine spiritual practice is that the tools you acquire through working on yourself are powerful and useful, which is why secular people are drawn to yoga/meditation as "just" a practice.
The problem here is that by removing the spiritual side, you get the all the tools but none of the restraint required to wield them gracefully.
One problem for those people is that the motivation is wrong unless there is right view. If you don’t realise samsara, and don’t realise that everything is on fire (see fire sermon) and wish to escape, then why are you practising? Unfortunately for a lot of secular people the answer is: because I had a crazy psychedelic trip and I just feel that something deeper is there that I must explore. And this causes all kinds of issues, because they have no idea how to separate any insight they might have had in their trip from straight up visuals or intense emotions. Further they are interested in the truth for the sake of the truth, and the Buddha explicitly said that such paths can lead to madness. It’s just very dangerous to go deep into Buddhist insight practise without having proper motivations or guidance.
The problem here is that by removing the spiritual side, you get the all the tools but none of the restraint required to wield them gracefully.