I agree on the cloud thing. Don't agree that "high performance disks, ballooning memory, live migrations, and isolated workloads" preclude from using k8s - you can still run it as base layer. You get some central configuration storage, machine management and some other niceties for free and you can push your VM-specific features into your application pod. In fact, that's how Google Cloud is designed (except they use Borg not k8s but same idea).
True! I love the idea of using K8s to orchestrate the running of VMs. With graceful shutdown and distributed storage, it makes it even more trivial to semi-live migrate VMs.
Are you aware of the limits? It must run as root and privileged?
In this scenario k8s is orchestrating the hypervisor,
not VMs themselves. Hypervisor then orchestrates VMs + network (eg OVS) + other supporting functions (logs shipping, etc) on each individual “worker” node. VM scheduling/migration component needs to be completely decoupled from k8s apiserver (but itself can still run as normal k8s deployment) bc scaling kube api with unbound users is challenging. And yes, hypervisor will need to run privileged but you can limit it to worker nodes only
