Our project is a content management system supporting several dozen of our websites. The development group started off small and in one location, and we dealt with a fairly standard coding/deployment strategy.
We coded off of trunk and insisted on a clean trunk. Every few days we would tag trunk and deploy to the test server. If all worked out, we'd deploy to Production and move on.
That worked well for a while until the team grew. We frequently faced situations where the revision that was tagged had problems that needed to be fixed before going to Production. While the developer responsible was working on those fixes, we had other developers committing changes to trunk. Once the original developer's fixes were complete, the new commits that were added would have to go along for the ride, further delaying the build because now there's additional validation that needs to be done.
In an attempt to rectify this, we created a separate trunk used strictly for releases. People would work in the main trunk, and then ask the project manager or development lead to merge their changes into the release trunk.
This worked for a while until the team got even bigger and more disjoint. We have teams of 3-5 people working in 4 geographic locations - some on the same components, others on different components with different priorities and release schedules. This was pretty much a full-time job and became a nightmare for the person managing the builds.
In an attempt to work around that, we started creating "release branches" off of whatever the latest Production tag was. People would commit to there ONLY what is ready to be tested and go to Production. Others would commit to trunk until it was their turn to merge. That took the burden of merging and resolving conflicts off of the build manager and on to the person owning the code.
This worked for about a week until we started having to do several "high priority emergency" releases. This effectively meant that we would:
- Create a branch off of the latest Production tag
- Add the emergency stuff to that branch
- Tag that branch and release to Production
- Merge all of the changes that were made in that branch into the regular "release branch" that is sitting in QA.
This is every day. Sometimes twice a day.
I've tried to relate this a bit to an open source project where there are developers all over the place who don't even know each other and they still seem to get by... but that comparison falls apart when new stable, tested, production-worthy builds are expected for "public" consumption several times a week (or day). If Firefox's daily build is a buggy mess, for example, at least users can go back to a previous version or use the latest stable release. That's not the case for our users. If our release is not perfect, they can't work.
Backstory completed, I now pose the question:
Given an environment where...
- Developers are all over the place and working on different components.
- Changes to some components can wait a week before being released, others can't wait even a day.
- The application is mission-critical and changes must be tested and stable before being released.
... what suggestions or alternative workflows can you recommend to promote a saner process where the majority of the burden is not on one person?