I never tried it, but I think it would go very poorly without a wake word of sorts.

HomeAssistant seems to natively support wake words, but I haven't looked into it yet. I simply use my smartwatch (Wear OS supports replacing Google Assistant with HomeAssistant's Assist functionality) to interact with the LLM