Processes and tools for testing large projects with multiple branches

Question

We make use of a significant number of branches (a fairly traditional mainline model) for development, and it's proved extremely effective as a way of organising things and keeping developers efficient on a large team. We have QA test development branches before pushing them back to the main line, which ensures the main line is always stable.

Now there are some interesting problems related specifically to testing. The most common one is this: suppose a tester encounters a bug during testing, that's already marked as fixed. Is that because the fix failed (in which case they should reopen the bug), or because the fix hasn't reached the branch being tested?

As perforce users, we're looking at solving these issues with perforce jobs. It's quite a "raw" tool though - it more or less provides the underlying functionality for this, but not an easy interface, particularly for testers to use. So I'm wondering if there are more user-friendly methods out there, or different approaches entirely (I don't think "avoid branching" is a practical answer for us in this case though!)

What are the best practices for performing effective QA on multiple branches? Are there any good tools out there that provide automation and support for these issues?

Answer 1

Sometimes testing is done once on the development branch, and again after merged into the release branch. But that is a process choice, not a requirement.

If you only release from the "mainline" branch, you might consider changing your QA process so that you only test after changes are merged into the release branch. And rely on the developer to test their changes pre/post merge. This is more typical.

With what you have now, it sounds like maybe you have one branch per developer (just a guess). And you submit bugs against each development branch. I would seriously consider changing this process. It is not maintainable and will be very challenging tracking bugs after merges.

As TrueWill suggested, setting up a continous integration would be a good idea. Supplement that with frequent (daily) builds. Then have QA test the builds.

Answer 2

Hey.
If qa finds bug in main branch, and it is fixed in dev branch then:

• let qa check that it is fixed in dev branch
• let qa comment bug in the main branch, but keep it opened
• after fix is released to main branch lest qa test it again it main branch
• if fix fixed it in the main, close bug in the main branch.

As for other situations try to workout scenarios like this (here you have piece of bug life cycle). Prepare scenarios for when to test and what to test, How communication qa-dev should look like. Write everything down. That will be your test process. Make QA follow it (and dev where appropriate). Probably you won't cover all situations right away, so improve your process with time.

As for big multi branch project. Maybe think about hiring test manager/test lead/ or whatever you want to call this role. Someone who will manage the qa work. Prepare test strategy. Prepare test plans. Coordinate qa work with dev team. Ensure proper communication. Additionally this person could manage QA work between different branches. In big complicated projects with parallel branches qa needs good management (as everybody else).

As for tools there may be few, but I worked only with HP Quality Center and HP Quick Test Pro. QTP is test automation software (record&playback, data driven, scripted - VB). QC is repository for Tests (both manual and automated), Defects, also can hold Requirments, Test Results. It can keep track of requirement-test coverage, and test-defect link (actualy test run-defect, but you can jump from test run to test).
Additionally you can define in it Cycles and Releases, so for example it can be Cycle per branch and, Release per code release (where each Release is related to given Cycle, so it would be clear to which branch code was released). Or you can map it differently.
Problem is, that software costs money. I mean really money. It is a commitment.