we have hundreds of clients most on k8s or openshift. Many of them are shifting to smaller team or division oriented clusters so that they can move at their own pace in consuming capabilities, tools, security practises, etc. It also simplifies distribution of operational expenses, yea there’s tools out there to help with that but it’s a whole lot easier to hand someone a sub-account in AWS and call it a day. Additionally it reduces your fault boundary. It isn’t as sexy as saying you have a 5000 node cluster with 100k+ pods running but it can make life a whole lot less stressful when it comes to upgrades and changes.
So basically, doing to infrastructure the microservices approach?
a) if you're going to intentionally make the trade-off to have higher costs in exchange for reducing ops load and simplifying administration within an account, where services make network calls across accounts, then adopting ECS/Fargate is probably a significant improvement over Kubernetes.
b) The underlying engineering / financial reality is still there and very much a leaky abstraction that will show up on your AWS bill. Cross-VPC, cross-region, and cross-AZ networking costs are very real overhead that you must consider; you either still have centralized network planning with shared VPCs and subnets or you basically decide to give up and let AWS send you the bill when teams create their own VPCs, their own subnets, etc.
c) setting up DNS / service discovery and network controls within a single cluster is simple. Doing so across account boundaries is not, and deciding to set up solutions like AWS Transit Gateway incur their own costs.
I'm sure it simplifies some things for some people, but Finance is going to get upset pretty quickly. Once you start to go down that path, it's very difficult to back out.
Everything’s benefits and tradeoffs. With this model there’s a clear ownership model. I’m not suggesting it’s right for everyone but it is a trend we’ve noticed with a number of clients over the last 2-3 years.
imo you've just traded one set of centralized costs (developing tools, security practices that work for all and educating people) for a distributed set of costs with everyone managing their own clusters and duplicating all this work. You've just pushed the cost onto other teams - which can be a fine approach for reducing the central teams stress, but is net more costly to the business. EDIT: Im not saying its necessarily a bad choice, just much more expensive.
It’s all swings and roundabouts has been since the broad adoption of computers. I expect in a few years time when everyone’s forgotten the pain points that drove them to decentralize there’ll be a big move to centralize again. Someone will get a gold star for coming up with the idea and being able to demonstrate the “cost savings”. Of course it’ll completely ignore the general disruption to all of the product teams as they adapt to the new world order but what’s a company without a little busy work.
