I went this route with code-server[1]. But later decided against it because some shortcuts don't work in the browser. Also installing and managing code-server on those VMs had started becoming a chore.

I am using remote-ssh instead. Hassle-free and works like charm.

Can't comment on sudo issue because I mostly log in using root.

[1]: https://github.com/cdr/code-server [2]: https://marketplace.visualstudio.com/items?itemName=ms-vscod...