I do contract devops for both physical servers (that includes me occasionally travelling to a data center), managed servers in a full service colo, and AWS, and the overall cost per server consistently ends up higher including all devops time for the AWS instance in my experience.
I love it when customers pick AWS (though I usually advice not to, unless they have very specific needs), as my billable hours are way higher for those clients, though it is annoying having to deal with the inevitable "why is my AWS bill so big?" after I'd told them exactly why it'd be expensive in the first place. This is particularly true with bandwidth heavy setups, where AWS charges tens of times more per TB transferred than e.g. Hetzner.
Often I even end up getting paid to help them get off AWS again down the line when they realise how expensive it is.
Overall the idea that managed servers and even bare metal colocated servers take up so much more operations time is just not what I experience. Even if it did, if you're even moderate successful it'd need to save you a crazy amount of operational time per server to make up for the price differences in the hosting.
Further my experience is that you don't need to over-provision much exactly because setting up hybrid setups where you spin up AWS instances (or any other cloud) to handle spikes works well. The trick is to treat managed servers exactly as cloud instances, apart from at most the initial hardware provisioning (though many hosting companies provides APIs for this so you can abstract away that too), so that it doesn't matter where you provision.
As for services like RDS etc., they're a very mixed bag. When they work for you, great, though they're expensive, but very often I end up with clients having to move off them because they need some plugin or other that isn't available. Very few of my clients in the end stay on them for very long. Their biggest benefit is to defer the initial setup of a self-managed cluster.
That doesn't mean there are no cases where AWS is the right choice - for starters having it there for traffic spikes is great (though we usually end up needing it very rarely), and if you need large batch jobs or other environments you spin up/down frequently, it may be cost effective. But it's extremely rare I see cases where it's cost effective for base load.
The typical argument then is that big companies wouldn't have picked them if it wasn't. But big companies don't pay the prices mere mortals pay. I know concrete examples of large negotiated discounts for a couple of larger companies, and they are steep.
> and I'd have to spend more money and time on auditors
That might be true, but most companies are not in a position where they need their IT infrastructure audited. This might very well be a niche that makes it worth using AWS.
> And I've haven't even started to think about securing the resulting systems to the same level I get for minimal effort from a major public cloud.
My experience is that getting security right on the public clouds is harder than bare metal. If you take the effort to do it properly, the end results can be good. But a lot of that is simplified a lot in a colo'ed environment by simply physically separating resources into different network segments and the like.
For people with very complex requirements, you might even get better results, but I could never agree with "minimal effort" - it's very common to see people badly misconfiguring their IAM setup for example, because it was too hard for them to figure out how to open up just the specific things they needed to open.
All our duelling anecdotes aside, you've picked out the one category I would most definitely not host on AWS: high constant outbound traffic sites. "Don't build your CDN on AWS", I can agree about that. Using public clouds for the spiky part of an otherwise predictable steady-state workload is also a great strategy. Also this:
> Their biggest benefit is to defer the initial setup of a self-managed cluster
And that, I think, is why (nearly) every startup, or indeed every new project, should begin on a public cloud. The capital waste of an incorrect server purchase can ruin a project and makes "fast failure" hard to stomach.
I don't move services away until an alternative is clearly both cheaper in 3yr NPV, and will not constrain future business opportunities. Except in cash cow operations where innovation has ceased, I rate the second criteria more important than the first and a compelling reason to stay on a public cloud.
I tend to favour starting with rented servers too, rather than purchases (or leases, rather) - whether that's on AWS, or rented by the month then becomes a much simpler comparison.
As for the CDN, we absolutely agree. I noted elsewhere that even if you host on AWS, if you have any kind of volume of outbound bandwidth use, you should probably get a CDN elsewhere whether or not you think you need a CDN. A good caching CDN can for many people cut their AWS bill dramatically without making you move anything else off AWS.
As mentioned, its traffic charges are as high as EC2/S3. CloudFront is useful if your goal is to reduce load on your EC2/S3 setup for reasons mostly other than cost, and want an all-AWS stack.
If cost is your reason for looking at a CDN, or a big part of it, CloudFront will do very little for you unless your pageviews are extremely costly in terms of compute relative to the amount of data returned, and said data is very cache-friendly.
That's very much a niche requirement. To date I've not come across a setup where CloudFront made sense cost-wise.
Good to know. I've noticed that NixOS hosts their binary package caches on CloudFront, which I assumed was because it's cost effective. Those URLs are completely content addressed so they're optimal for CDN caching. Is there a significantly better alternative to S3+CloudFront? I'm pretty influenced by brand names in this situation because there are so many random hosting companies out there.
You can keep S3 if your cache hit rates are good enough and still save a lot. S3 is great for the storage for e.g. durability, as long as you can avoid serving up too much of your content front it.
To be honest I rarely use external CDNs and instead "roll my own" with a variety of providers as with cloud providers + geoip enabled DNS you can get 90% of the benefit at very low rates, but in terms of "brand name recognition" MaxCDN is worth checking. It's not nearly the maximum saving you can get, but it can be substantially cheaper than CloudFront.
Traffic pricing for CloudFront is similar to standard traffic pricing on AWS (e.g. for EC2 instances).
They charge you also per request and user origin a bit extra, so in some cases maybe even more expensive.
For the life of me, I can't figure out why hackernews is still always talking about AWS. Every single time I've tried to look at it for a use case of mine it's the most expensive possible option. Cloud? Sure, I see the benefit and plan to go on the cloud (probably Azure, IBM or so). AWS? I don't get it.
I love it when customers pick AWS (though I usually advice not to, unless they have very specific needs), as my billable hours are way higher for those clients, though it is annoying having to deal with the inevitable "why is my AWS bill so big?" after I'd told them exactly why it'd be expensive in the first place. This is particularly true with bandwidth heavy setups, where AWS charges tens of times more per TB transferred than e.g. Hetzner.
Often I even end up getting paid to help them get off AWS again down the line when they realise how expensive it is.
Overall the idea that managed servers and even bare metal colocated servers take up so much more operations time is just not what I experience. Even if it did, if you're even moderate successful it'd need to save you a crazy amount of operational time per server to make up for the price differences in the hosting.
Further my experience is that you don't need to over-provision much exactly because setting up hybrid setups where you spin up AWS instances (or any other cloud) to handle spikes works well. The trick is to treat managed servers exactly as cloud instances, apart from at most the initial hardware provisioning (though many hosting companies provides APIs for this so you can abstract away that too), so that it doesn't matter where you provision.
As for services like RDS etc., they're a very mixed bag. When they work for you, great, though they're expensive, but very often I end up with clients having to move off them because they need some plugin or other that isn't available. Very few of my clients in the end stay on them for very long. Their biggest benefit is to defer the initial setup of a self-managed cluster.
That doesn't mean there are no cases where AWS is the right choice - for starters having it there for traffic spikes is great (though we usually end up needing it very rarely), and if you need large batch jobs or other environments you spin up/down frequently, it may be cost effective. But it's extremely rare I see cases where it's cost effective for base load.
The typical argument then is that big companies wouldn't have picked them if it wasn't. But big companies don't pay the prices mere mortals pay. I know concrete examples of large negotiated discounts for a couple of larger companies, and they are steep.
> and I'd have to spend more money and time on auditors
That might be true, but most companies are not in a position where they need their IT infrastructure audited. This might very well be a niche that makes it worth using AWS.
> And I've haven't even started to think about securing the resulting systems to the same level I get for minimal effort from a major public cloud.
My experience is that getting security right on the public clouds is harder than bare metal. If you take the effort to do it properly, the end results can be good. But a lot of that is simplified a lot in a colo'ed environment by simply physically separating resources into different network segments and the like.
For people with very complex requirements, you might even get better results, but I could never agree with "minimal effort" - it's very common to see people badly misconfiguring their IAM setup for example, because it was too hard for them to figure out how to open up just the specific things they needed to open.