tracking deployments across many teams to an environment, how do the pros do it?

Question

The scenario: we had a break in one of our environments. Turns out that someone deployed a new version of a view that broke someone else's code. Took quite some time to track it down. Got me to wondering though.

Is there a best practice for managing deployments across many teams? What is your opinion of a good practice for managing this?

My idea (half formed, as you'll see) is this: have a deployment database. Anyone who does a deploy is required to make an entry in the deploy database: where the new code resides, what environment that code went to, who authorized it and when it happened. First problem: enforcement. Second problem: adoption. Third problem: backward compatibility.

Answer 1

Is there a best practice for managing deployments across many teams? What is your opinion of a good practice for managing this?

The easiest way is to do deployments as part of your continuous integration process. Either manually or automatically, builds can be deployed at regular intervals or upon reaching some threshold.

Since most CI servers support some level of auditing, it will be trivial to see what changes were introduced in a particular deployment that might be breaking things. This solves a number of problems at once:

• Enforcement: Only people who can deploy a build will be able to deploy to production this way. Access is controlled by CI.

• Adoption: Trivial. Nobody has to do anything other than click a button to initiate the deployment.

• Backward compatibility: Trickier. You would need to define more clearly what requirements you need to support, but chances are good this gets knocked out too.

Answer 2

I've run into a similar situation as you've described. The way we managed to fix it is much like you said. We put a large emphasis on documentation and approval. Every (non-trivial) change in code must be documented, tested, and approved by another person before that change can go into development. We also have the luxury of using TFS for scheduling builds and source control.

If this is too costly of an option for you, then I suggest creating a program that will send out the updates as well as creating a backup folder containing the previous version. If the updates cause a problem, rolling back would be as easy as going to the backup folder placing the previous version back into the production environment.

Hope this helps.