Be mean about K8s, it can take it. I don't think any of the criticism of it is unreasonable. It's a huge, convoluted beast. To me, K8s makes most sense in terms of creating a swiss army knife that lots of people know.
You can do far better than K8s for specialised cases like yours and/or with people who has the right skills. But for a lot of people with simpler needs it's better to just stick to what is easy to hire for than look for an optimal solution.
It ain't that. It's that k8s-aas is available everywhere (and if it's not, then it'd be soon). So knowledge is transferable.
And most don't want to, and shouldn't setup their own orchestrator. It's painful, cumbersome, and a waste of money, where you could get a k8s-aas everywhere.
Fly are the rare example of a company that does need to run their own orchestration, and indeed, k8s doesn't fit them (it rarely does, in such cases. Unless you sell k8s-aas).
I mostly agree with you, but would qualify that by saying that writing a specialized orchestrator is not that hard for a lot of cases. I have done so a couple of times.
Lots of decisions will likely end up similar to K8S - e.g. for one I ended up putting SkyDNS/CoreDNS for service discovery on top of Etcd and using a Nginx handler to route ingress - when we first upgraded our (many years old) system to doing that, K8S was not yet widely used and I was unaware K8S did the same.
But if you have a specific use case and understand it well, it can be worth it.
However, even if you have a situation like that where it's genuinely cost effective for you to write your own, a big reason why K8S might still be a better choice is that you can hire people who know K8S - if you build your own you need to worry about knowledge transfer for something very critical to your operation.
I think writing orchestrators is fun, yet I still think most people shouldn't do so for production (but write a toy one as a learning experience anyway)
A lot of things are simple to write a naive implementation of, and sometime they useful to solve a small problem, even if just to prove this is the way to go, and its worth spending on a bigger, generalized and maybe more expensive to maintain in the short term solution.
I myself have written smol messaging queues, metrics collectors, mesh router, and a few more services that once it was agreed that they had a place in our systems, we migrated to NIH software, so we could enjoy the shoulders of giants (or at least buy support for it, rather than bus factoring it on myself and another colleague).
A production quality generic orchestrator is a lot of work because it needs to handle a vast amount of different scenarios (e.g. the list of volume types supported by k8s is vast). An orchestrator tailored to specific workloads and circumstances is often not for someone with a reasonable amount of devops experience. Hence about 1k lines of straightforward code for one of the production specialised orchestrators I wrote.
You don't avoid having to account for bus factoring anything by running k8s. It's dangerous to assume so. A k8s config for a similar sized fleet would've been many times the size of the custom orchestrator (heck, the k8s config for my home test k8s-cluster is bigger than that orchestrator), and you still need to train people on the architecture. In a custom orchestrator - at least the ones I've written - it's natural to encode a lot of assumptions about architectural choices in the code, and so all our devs could read the code and know the architecture if needed, without any presumption of needing to train them on tools they were unfamiliar with.
The reason I still agree most people should just run k8s is that it's easier to hire people who understand k8s than to hire people experienced to do a good job developing a custom setup in the first place. Our devs could trivially read and understand and modify the orchestrator once it was written as needed, but that's not the same as assuming they had the experience to make the architectural choices encoded in it in the first place.
K8s is a suitable straight-jacket to stop people from making really stupid choices, and so it can substitute for a lot of expensive and rare experience, and that's good. There are still plenty of places where there's room to do custom work.
EDIT: To add, I think a useful guideline to this is: Are you writing generic code that there's an existing solution to, or is your code effectively defining policy, architecture or config in a compact way? If the former, I'd agree you need a really good reason for doing so. In the latter, much less so. E.g my custom orchestrators have been small because 90% of the functionality we needed was there in other components - often the same components k8s ended up using - but where k8s needs to provide generic apis to allow applying a generic configuration, we could encode a specific architecture in the way we structured the orchestration. The availability of those components, incidentally, has gotten better thanks to the presence of the big, complex generic orchestration solutions, making it easier than ever to do custom orchestration when it works for you.
Surprisingly often "crappy but sufficient and possible to hire for" beats good.
Hiring good devops people with enough experience of alternatives to be able to pick an alternative or build something custom that is good enough to be worth it is hard.
It only became fashionable to call k8s crappy because it effectively won the orchestration wars. In reality it's quite good, and it changed the game for everyone. The hype for it went overboard a good bit so some swing back is justified. It's too complex for many use cases. The API surface isn't perfect but it's rather consistent and well defined, the project's codebase is mostly a good example of style at this size. Really I think the authors of k8s should be proud of what they built.
I think the complexity is the main source of people thinking it's "crappy". It certainly my main issue. E.g. I've built orchestrators from scratch smaller than some k8s configs I've worked on. However they were special purpose - they worked well for the specific type of loads, and for that they were suitable because their tiny size meant they were easy to tweak, but they'd have been totally inappropriate as a general-purpose alternative to k8s.
And so even though I don't like it much, I think it's fine to default to k8s for people (including at my current day job), and the default will always get a lot of crap because the default choice will get pressure to include all kinds of functionality that makes it more complicated in order to support all of the very diverse workloads that don't - and likely never will - justify a custom solution.
You can do far better than K8s for specialised cases like yours and/or with people who has the right skills. But for a lot of people with simpler needs it's better to just stick to what is easy to hire for than look for an optimal solution.