Release/Deployment/Maintenance process for a web service

Question

Hi all,

I'm doing research on some non-brittle release process techniques in a 24x7x365 web service environment. What are some of your favorite methods for releasing code from a development environment to a critical-production environment. This includes not only code changes -- but also database schema changes. Let's assume that everything has been tested and given the thumbs up in the ecosystem of a development environment.

What steps do you take next to streamline it to production? Redirect traffic to a downtime cluster and run some automated scripts? If so, what do your scripts do? If not scripts, what kind of routine are you fond of?

Looking for only technical insight. Please no "give customers ample warning" type responses.

Thanks!

Answer 1

There are a couple of principles that I think are important to ensure:

1. Release only from source control system (either a dedicated branch or a tag on mainline). No possibility of releasing directly from working directory.
2. Make push-to-production and push-to-staging processes utterly automated (including schema updates)
3. Have automated rollback script to hand, and don't be afraid to use it if anything seems wrong.
4. All database schema changes are done with incremental roll-forward/rollback capable tool (like ActiveRecord or Liquibase). Requiring full DB restores should be rare.
5. (for really large clusters) Do rolling upgrades; use two load balancers, one for the version N release and one for the N+1 release. Test continuously as additional nodes join N+1 group.

Answer 2

Denis Hennessy wrote a fine summary. As for tools, have a look at Capistrano.