Whisper will hallucinate on audio segments that don't have any speech. VAD mitigates that. Expect worse results without it, especially on non-English audio.