Having a staging server doesn't only help with capacity problems, but more so with deployment problems and differences in the environment. You might be running 32 Bit on your server but 64 Bit on your dev box. Or prod uses Amazon RDS, but locally you just have MySQL on the same box. And oh suprise, now your mysql gem doesn't want to be built. Or you ran each migration on your dev box separately as you build them and when you run them all at once during your prod deploy, things blow up. That's what a good staging environment should protect you from. And for all of that to be a problem, 2 users a minute are more than bad enough. You don't need to be Facebook for that.
Even better, if you for example host your app on heroku, you get you staging and demo env for free in less than 30 minutes!
Patrick's reference to "capacity problems" was in regards to deciding whether to clear memcached. In a large scale deployment, such as Facebook, clearing the caches could overwhelm the servers. Patrick is small-scale enough to not worry about that, and prefers clearing the cache to avoid stale-cache-related bugs.
I do agree regarding how nice it is to use Heroku. My only issue with them is that they only support Ruby and Node.js, so I need to take my Clojure applications elsewhere.
I also recommend to have a demo server, to which you can push your latest changes without thinking twice. This way, demoing new features to the customer or testing your deployment script does not change the staging box.
This way you can have your staging deploy be a much more realistic test run of your production deploy. You only push to staging, when you are about to push to production. Otherwise you might get state differences between the two like outstanding migrations that need to be run on one server but not on the other one. Typically things like that beak you neck during deployment to prod, so you want to test that. But you still want to have a faster way of getting features vetted by your customer. So you should have demo and staging.
One of the nice things about being on the cloooooooooooud is that, if you've got your infrastructure managed right, a developer who wants a sandbox -- for any reason -- should be able to get one with about as much difficulty as getting office paper, and be able to bin it with about as much regret.
If you're hosting with Amazon and your application data is on its own EBS (block storage service) volume cloning your environment is easy. You can snapshot the volume, create a new volume from the snapshot, and spin up a new server (you automated provisioning, right?). That way you get a near exact replica of your environment with not too much work.
This is what we have with an Oracle product:
Vanilla - no sample or legacy data, no dev
Sandbox - oracle sample data, no dev
Conversion - legacy data, no dev
Dev - "scrambled" legacy data, dev
Config - functional team environment setup
Test - config and dev together, unscrambled user data
Test2 - I have no idea why this here, but it is :)
Training - for training users, selected legacy data and dev
PreProd - everything after test and before prod
Prod
Given that each environment has an app server, a web server, two or more batch servers and a database, I can vouch that a fubarred deployment workflow is hell. You end up with no baseline to tell if your process is bombing due to configuration or code.
I always wonder were amazon video on demand is in these discussions. I find that service much more comfortable to use and you can redownload your videos or even watch them in the browser. I wonder why this is not getting more attention
