Continous integration and QA

Question

in my project, we are setting up continous integration environment and as part of this process, propose simultaneous fixing of defects during the QA test cycles.

What is the generally practiced process when it to releasing this into the QA environment. Are these fixes immediately deployed into the QA environment (after itnegration testing) or are they accumulated till the completion of the current test cycle.

Answer 1

Giving yourself a constantly moving target is really difficult. We tend to batch fixes up before deploying to QA- usually about a daily QA deploy. QA takes the output from the overnight build first thing in the morning, and "as needed" if a really critical bug is blocking a lot of testing.

CI is more a basic code-quality benchmark (eg, it builds, it passes unit/smoke tests)- don't feel like QA needs to take every build coming out of CI.

Answer 2

In I understand correctly, you are asking what is the duration of the QA cycle in a project that has a Continuous Integration cycle?

We use a daily QA cycle. The night build, if it succeeded, is available for testing during the next day.

Answer 3

nitzmahone's answer is great advice for balancing the more frequent builds you'll get from a CI system with QA's need to have a known target for testing.

You can take advantage of continuous integration to get additional testing on top of the unit / smoke tests that are run as part of each build. Here are a couple of things I've done:

• Set up a job in your CI system that runs a scheduled job (e.g. daily, as opposed to being triggered by source code changes) to do a performance test on your system, and log the results. This way you can track performance over time and spot any changes that are having an adverse effect on performance.
• Have your CI build job auto-deploy your system if the build and unit tests are successful, and have an IT monitor such as Nagios checking the deployed system. This acts as a quick and dirty system test that can often find bugs that aren't caught by the unit tests. I wrote a blog post that you might find useful if you're interested in this type of test.

Answer 4

Are these fixes immediately deployed into the QA environment (after integration testing) or are they accumulated till the completion of the current test cycle.

It depends. If an issue is blocking and doesn't allow testers to run more tests, to finish a whole test plan (i.e. to do their job), then it might be obviously required to release the fix immediately. If an issue is not a blocking one, then it might be preferable to queue the fix and make it available in the next release. But this requires lots of administrative work (logging the issue, annotating the test case, etc...).

Now, if QA occurs very early during development (i.e. if you are not using a very sequential development cycle), if testers are working closely with developers, then it might be good to fix an issue as soon as it is discovered and to even avoid creating an inventory of bugs (big waste).

Answer 5

At the beginning of test cycles I tend to only take new builds if there are blocking bugs that are resolved. This allows my team to avoid the thrash caused by new builds and surprise regressions. The early part of a release is often spent understanding how the functionality was actually implemented and if the product meets a minimum set of acceptance criteria.

In the middle of a testing cycle I accept builds more often in order to ensure fixes have as much exposure as possible and to identify improperly fixed bugs as soon as possible. This is usually daily except in environments where we are running long term stress or performance tests.

As the release approaches and I exert more control over what bugs are fixed (ie: the current release is branched and we have a more strict code line policy) I'll continue to take builds only as we find release blocking bugs. At this time builds are often referred to as beta or release candidates.