I see the shade thrown at k8s... but honestly I don't know how much of it is truly deserved.
k8s is complex not unnecessarily, but because k8s is solving a large host of problems. It isn't JUST solving the problem of "what should be running where". It's solving problems like "how many instances should be where? How do I know what is good and what isn't? How do I route from instance A to instance b? How do I flag when a problem happens? How do I fix problems when they happen? How do I provide access to a shared resource or filesystem?"
It's doing a whole host of things that are often ignored by shade throwers.
I'm open to any solution that's actually simpler, but I'll bet you that by the time you've reached feature parity, you end up with the same complex mess.
The main critique I'd throw at k8s isn't that it's complex, it's that there are too many options to do the same thing.
I think part of the shade throwing is k8s has a high lower bound of scale/complexity "entry fee" where is actually makes sense. If your scale/complexity envelope is below that lower bound, you're fighting k8s, wasting time, or wasting resources.
Unfortunately unless you've got a lot of k8s experience that scale/complexity lower bound isn't super obvious. It's also possible to have your scale/complexity accelerate from "k8s isn't worthwhile" to "oh shit get me some k8s" pretty quickly without obvious signs. That just compounds the TMTOWTDI choice paralysis problems.
So you get people that choose k8s when it doesn't make sense and have a bad time and then throw shade. They didn't know ahead of time it wouldn't make sense and only learned through the experience. There's a lot of projects like k8s that don't advertise their sharp edges or entry fee very well.
> I think part of the shade throwing is k8s has a high lower bound of scale/complexity "entry fee" where is actually makes sense. If your scale/complexity envelope is below that lower bound, you're fighting k8s, wasting time, or wasting resources.
Maybe compared to Heroku or similar, but compared to a world where you're managing more than a couple of VMs I think Kubernetes becomes compelling quickly. Specifically, when people think about VMs they seem to forget all of the stuff that goes into getting VMs working which largely comes with cloud-provider managed Kubernetes (especially if you install a couple of handy operators like cert-manager and external-dns): instance profiles, AMIs, auto-scaling groups, key management, cert management, DNS records, init scripts, infra as code, ssh configuration, log exfiltration, monitoring, process management, etc. And then there's training new employees to understand your bespoke system versus hiring employees who know Kubernetes or training them with the ample training material. Similarly, when you have a problem with your bespoke system, how much work will it be to Google it versus a standard Kubernetes error?
Also, Kubernetes is really new and it is getting better at a rapid pace, so when you're making the "Kubernetes vs X" calculation, consider the trend: where will each technology be in a few years. Consider how little work you would have to do to get the benefits from Kubernetes vs building those improvements yourself on your bespoke system.
Honestly, the non-k8s cloud software is also getting excellent. When I have a new app that I can't containerize (network proxies mostly) I can modify my standard terraform pretty quickly and get multi-AZ, customized AMIs, per-app user-data.sh, restart on failures, etc. with private certs and our suite of required IPS daemons, etc. It's way better than pre-cloud things. K8s seems also good for larger scale and where you have a bunch of PD teams wanting to deploy stuff with people that can generate all the YAML/annotations etc. If your deploy #s scale with the number of people that can do it, then k8s works awesomely. If you have just 1 person doing a bunch of stuff, simpler things can let that 1 person manage and create a lot of compute in the cloud.
Are you referring to instances of your application, or EC2 instances? If instances of your application, in my experience it doesn't really do much for you unless you are willing to waste compute resources. It takes a lot of dailing in to effectively colocate multiple pods and maximize your resource utilization. If you're referring to EC2 instances, well AWS autoscaling does that for you.
Amazon and other cloud providers have the advantage of years of tuning their virtual machine deployment strategies to provide maximum insulation from disruptive neighbors. If you are running your own Kubernetes installation, you have to figure it out yourself.
> How do I know what is good and what isn't?
Autoscaling w/ a load balancer does this trivially with a health check, and it's also self-healing.
> How do I route from instance A to instance b?
You don't have to know or care about this if you're in a simple VPC. If you are in multiple VPCs or a more complex single VPC setup, you have to figure it out anyway because Kubernetes isn't magic.
> How do I flag when a problem happens?
Probably a dedicated service that does some monitoring, which as far as I know is still standard practice for the industry. Kubernetes doesn't make that go away.
> How do I fix problems when they happen?
This is such a generic question that I'm not sure how you felt it could be included. Kubernetes isn't magic, your stuff doesn't always just magically work because Kubernetes is running underneath it.
> How do I provide access to a shared resource or filesystem?
Amazon EFS is one way. It works fine. Ideally you are not using EFS and prefer something like S3, if that meets your needs.
> It's doing a whole host of things that are often ignored by shade throwers.
I don't think they're ignored, I think that you assume they are because they are because those things aren't talked about. They aren't talked about because they aren't an issue with Kubernetes.
The problem with Kubernetes is that it is a massively complex system that needs to be understood by its administrators. The problem it solves overlaps nearly entirely with existing solutions that it depends on. And it introduces its own set of issues via complexity and the breakneck pace of development.
You don't get to just ignore the underlying cloud provider technology that Kubernetes is interfacing with just because it abstracts those away. You have to be able to diagnose and respond to cloud provider issues _in addition_ to those that might be Kubernetes-centric.
So yes, Kubernetes does solve some problems. Do the problems it solves outweigh the problems it introduces? I am not sure about that. My experience to Kubernetes is limited to troubleshooting issues with Kubernetes ~1.6, which we got rid of because we regularly ran into annoying problems. Things like:
* We scaled up and then back down, and now there are multiple nodes running 1 pod and wasting most of their compute resources.
* Kubernetes would try to add routes to a route table that was full, and attempts to route traffic to new pods would fail.
* The local disk of a node would fill up because of one bad actor and impact multiple services.
At my workplace, we build AMIs that bake-in their Docker image and run the Docker container when the instance launches. There are some additional things we had to take on because of that, but the total complexity is far less than what Kubernetes brings. Additionally, we have the side benefit of being insulated from Docker Hub outages.
k8s is complex not unnecessarily, but because k8s is solving a large host of problems. It isn't JUST solving the problem of "what should be running where". It's solving problems like "how many instances should be where? How do I know what is good and what isn't? How do I route from instance A to instance b? How do I flag when a problem happens? How do I fix problems when they happen? How do I provide access to a shared resource or filesystem?"
It's doing a whole host of things that are often ignored by shade throwers.
I'm open to any solution that's actually simpler, but I'll bet you that by the time you've reached feature parity, you end up with the same complex mess.
The main critique I'd throw at k8s isn't that it's complex, it's that there are too many options to do the same thing.