I was recently setting up production env and thinking of deployment process for the new web app I'm building. A few of my experiences/thoughs (my app is on a single VPS - test/staging and production servers are separate systems):
Patrick's suggestion to keep a log with all the setup steps for the production was already a life saver. Two days ago I remembered that my server was running 32bit code and I was going to run MongoDB on it. Whoops. Complete reinstall in 48min, worry-free.
I'm keen on git, and plan to use it to deploy to production. I actually rsync to test/staging server doing development[0] (to avoid having to commit knowingly broken code just to be able to deploy on testing/staging server), but I use git to manage the code, and to deploy it. I have a clone repo on the production server with a production branch, which has a few commits on top of master, in which I committed the production-specific configuration.
Deploying on production does roughly:
1. check out the latest code from master
2. rebase production on top of master in a temporary repo to catch any rebase problems (because I don't want to merge master into production)
3. run unit tests in a temporary repo (sadly my tests only test the backend, not the web ui; I plan to improve in 2011 :)
4. rebase real production repo
5. make a new tag (for almost-one-click rollbacks)
6. restart whatever services need to be restarted
This is automated by a simple shell script which aborts at the first sign of trouble.
Regarding the 3rd party packages versioning: I use system packages wherever available. I don't have automatic updates though, and I don't use a system having rolling updates (I'm on Ubuntu Maverick). I had to manually rebuild two packages: nginx (to include upload module) and Tornado web server (the one in Maverick is too old for my purposes). This was pretty straightforward, and I've recorded the exact steps in my server setup log.
[0] my app involves callbacks from external services, so I can't test it on my laptop; my development workflow is "save in editor, rsync, see whether it works", with services in debug mode reloading themselves as soon as change is detected.
> I have a clone repo on the production server with a production branch
Do you protect the dot-files from being accessed via the web? I was at a security conference recently and a speaker mentioned a number of companies had source code accessible over the web because they served directly from a repo and didn't block accessing the VCS files.
Patrick's suggestion to keep a log with all the setup steps for the production was already a life saver. Two days ago I remembered that my server was running 32bit code and I was going to run MongoDB on it. Whoops. Complete reinstall in 48min, worry-free.
I'm keen on git, and plan to use it to deploy to production. I actually rsync to test/staging server doing development[0] (to avoid having to commit knowingly broken code just to be able to deploy on testing/staging server), but I use git to manage the code, and to deploy it. I have a clone repo on the production server with a production branch, which has a few commits on top of master, in which I committed the production-specific configuration.
Deploying on production does roughly: 1. check out the latest code from master 2. rebase production on top of master in a temporary repo to catch any rebase problems (because I don't want to merge master into production) 3. run unit tests in a temporary repo (sadly my tests only test the backend, not the web ui; I plan to improve in 2011 :) 4. rebase real production repo 5. make a new tag (for almost-one-click rollbacks) 6. restart whatever services need to be restarted
This is automated by a simple shell script which aborts at the first sign of trouble.
Regarding the 3rd party packages versioning: I use system packages wherever available. I don't have automatic updates though, and I don't use a system having rolling updates (I'm on Ubuntu Maverick). I had to manually rebuild two packages: nginx (to include upload module) and Tornado web server (the one in Maverick is too old for my purposes). This was pretty straightforward, and I've recorded the exact steps in my server setup log.
[0] my app involves callbacks from external services, so I can't test it on my laptop; my development workflow is "save in editor, rsync, see whether it works", with services in debug mode reloading themselves as soon as change is detected.