Accidentally released code to live. How to prevent happening again?

Question

We had an incident recently where some code got released to live that was not scheduled to be released.

It had obviously been checked into the trunk. Which is fine i guess as you want to 'check in early, check in often'.

However in this instance it was not supposed to be released in the next release.

What kind of checks / strategy / process can be put in place to avoid code being released to live prematurely.

It seems to me even with Continuous Integration and Unit Tests this is a human procedure issue?

-- Lee

Answer 1

The problem obviously isn't with checking the code to the repository. You have two problems here:

1) Any code supposed to go live must have a special version tag or branch or whatever depending on the source control you use. So code going live never gets confused with code in development.

2) Who's the moron who puts untested code live anyway? There is a SEVERE lack of communication going on if the person who put it live thought your in-development code was production-ready.

Answer 2

Amend your integration procedures.

If "going live" means someone executing some batch script - don't be surprised if this will happen again.

Also, consider branching. A common example might be to use the trunk for development, a separate branch for testing (say, gets merged once a week), and a final branch (from the aforementioned testing branch) for RTC.

This branch, before being deployed to production, should be tested thoroughly.

Answer 3

You should have different branches if you source control software allows such thing.

In this scenario, you are going to have a person who is responsable for merging the code that already met the quality bar from the main branch into the production branch.

---

UPDATE: Although product specific, the guide available at TFS 2008 Branching Guide 2.0 has a lot of guidelines that can be applied to other source control software that have the capability of creating branches.

Answer 4

Don't build to production from the trunk - manually merge tested trunk code into a production branch and go to live from that. Or as the other answers say, use whatever amount of branches and steps in the testing procedure that fits your needs.

Also, code changes that takes more than a day or so should generally be made in a separate feature branch until it's done.

Answer 5

What kind of checks / strategy / process can be put in place to avoid code being released to live prematurely.

I would say any process that does not have checking in into the trunk as a habitual development ritual, which means any development model except for cowboy coding.

Let developers check in early and often into their feature branches and merge those into the trunk when the time has come.

Answer 6

To continue the discussion of branching, it's the way to go to keep the version-handling structured.

We use the trunk as the main-branch, and when we reach a certain point in the development cycle where we are not allowed to commit anymore features (only bugfixes), we branch a NEW branch for that release (instead of going through error-prone merging), and that branch is well tested before we build the release from it. Of course for this to work, every programmer need to keep the commits clean when the feature-freeze date is approaching.

Answer 7

We built a release manager that works with subversion. www.ReleaseManager.com So we can control what is released by Issue Number (or Bug Number) and we have control so we can pull things out of the release when needed. Looking for beta sites now :)

Answer 8

It seems to me even with Continuous Integration and Unit Tests this is a human procedure issue?

Indeed! However you should be able to get some support from your infrastructure to support the human side of your process. When you're going to do a release you should be able to easily see all the commits that would be part of it and all of the related issues. This is the reporting side of continuous integration. (I'd say there are four elements (pdf): building, deploying, testing and reporting.)