It's not even only that, when you switch to another worktree you catch your language server with its pants down, and some tooling e.g. test watchers are bugging out as well.

I just always found it cleaner to have several directories on the rare occasion I need to work on several branches almost in parallel.

And let's face it, most of us don't work in a repo that is 100GB or above. Even if it was a 10GB I wouldn't care one bit and can easily have 20 copies of it temporarily.