> but the real problem is the mandatory node draining that causes downtime/disruption.
> ...
> In theory, there's nothing stopping you from just updating the kubelet binary on every node.
I am pretty sure Kubernetes itself does not mandate node draining. I have been doing upgrade for bare metal cluster for years, and like you said, it's mostly just replacing kubelet binary and bounce.
However, I do understand in public cloud it's usually recommended to perform a node rolling update instead of modifying online nodes in place. Actually, I prefer this way because of the benefits of immutable infrastructure. The downtime is unfortunately, but so far I have been enjoying working with devs to better designing reliable apps to tolerate node issues like this.
I am pretty sure Kubernetes itself does not mandate node draining. I have been doing upgrade for bare metal cluster for years, and like you said, it's mostly just replacing kubelet binary and bounce.
However, I do understand in public cloud it's usually recommended to perform a node rolling update instead of modifying online nodes in place. Actually, I prefer this way because of the benefits of immutable infrastructure. The downtime is unfortunately, but so far I have been enjoying working with devs to better designing reliable apps to tolerate node issues like this.