Need advice or pointers on Release Management Strategies

Question

I look after an internal web based (Java, JSP, Mediasurface, etc.) system that is in constant use (24/5).

Users raise tickets for enhancements, bug fixes and other business changes. These issues are signed off individually and assigned to one of three or four developers.

Once the issue is complete it is built and the code only committed to SVN. The changed files (templates, html, classes, jsp) are then copied to a dev server and committed to a different repository from where they are checked out to the UAT server for testing. (this often requires the Tomcat service to be restarted and occasionally the Mediasurface service as well).

The users then test and either reject or approve the release. If approved the edited files are checked out to the Live server and the same process as with UAT undertaken.

If rejected the developer makes the relevant changes and starts the release process again.

This is all done manually without much control. Where different developers are working on similar files, changes sometimes get overwritten by builds done on out of sync code in other cases changes in UAT are moved to live in error as they are mixed up in files associated with a signed off release.

I would like to move this to a more controlled and automated process where all source code and output files are held in SVN and releases to Dev, UAT and Live managed by a CI system (We have TeamCity in house for our .NET applications).

My question is on how to manage the releases of multiple changes where some will be signed off and moved on and others rejected and returned to the developer. The changes may be on overlapping files and simply merging each release in to a Release Branch means that the rejected changes would have to be backed out of the branch.

Is there a way to manage this using SVN and CI or will I simply have to live with the current system.

Answer 1

It seems to me that rolling back the changes that are pushed into the Release branch seems to me a normal practice in your case, which I think might not be appropriate.

One thing that I find bit surprising is that why would users test changes and then either reject or approve the release ? IMHO testing of the changes and making sure that it fixes the issues raised by the users should be the task of a test team. The users are not testers. If there is a test team you can have a Testing (say) branch where the fixes/changes are pushed to, and the test team uses that branch for testing and qualifying the changes. Once the changes are qualified by the test team you can push the changes to the Release Branch (say) and do deployment from there. Even with this approach there will be chances that users will find issues with the changes made, but with the test team doing the internal testing the need for rollback would become an exception rather than the norm.

Answer 2

We have something similar going on over here.

But, we may have multiple changes between releases. Each change is done in main branch if it is done within a day. Otherwise it gets its own branch. When the change is complete it is merged to mainline and a new release-branch is made.

Overe here we actually do not go directly from main to release. We have a test branch in between.

If that test is OK, then it is branched into a new production branch, so we have 3 level deep branching, at the most. Many branches, but it keeps it simple to know what has changed since the last release, and keep track of which changes that should be in which environment (dev, test, prod).

Downside: Suppose you have something at test that must be kept there for a long time. You in between do many changes and do many test releases. You then need to merge the change from "test" to "test" so to speak. That is, it is possible to do a branch directly from main, but it would be easier to branch from test to test, and only accept a single change into the new test.

But, it is not possible to do this kind of cross merging in subversion AFAIK. TFS is just as bad in that area. I have been thinking about mercurial.. Edit: well, there is baseless merge in TFS, but it is kind of ugly and not supported in the VS IDE.

Answer 3

I am thinking about feature branching, not something I would attempt using Subversion.

Basically you create a branch for each issue. This branch you can build, deploy and have tested by users or testers. If the fix is not accepted you can continue improving the fix on the branch.

If the fix does get accepted you would still need to integrate it with any other fixes or pending changes before you can really release it any shape or form. You can go two routes here, you can integrate the fix on the version that is currently in production to create a 'maintenance release' or you can just merge the 'fix branch' with the mainline/trunk for it to get released together with whatever next release of the trunk. When working with a maintenance releases you need to be extra cautious and remember to merge the changes to the trunk at some point after acceptance or having gotten OK from testers. If not you may suffer from re-occurring bugs when the trunk gets released later on.

All forms of integration may warrant further testing; the fix might have been OK in isolation but perhaps doesn't perform too well when integrated with all the newest changes to the system. If the integration involved merges and such there may have been errors there also. With this in mind, I would stick to negotiating releases. Testers should test releases, not individual fixes in isolation.

Finding and fixing an issue doesn't always mean it has to be deployed instantly. Sometimes you can negotiate with your customer; "Is it OK if we release this bug-fix together with our next regular release?".

The whole bookkeeping and administration of chances, hot-fixes and out of band releases gets overly complex very quickly, I'd avoid it as much as possible. If you have 4 fixes and 1 is not OK yet and the other 3 can wait... go for that route.