Q/A, Release builds vs Debug builds, and Assertions

Question

Just curious:

When you release software builds to Q/A, do you prefer to always use the "RELEASE" version, or do you sometimes use the DEBUG version?

Here's my conundrum: We like to use Asserts to trap for conditions that should never happen.

On the one hand, it might be useful for Q/A to test our software with the Assertions enabled, so that if they can create a scenario that triggers an assertion, they can report it to us.

On the other hand, there is always the risk that a developer has coded an assertion in a way such that it changes the behavior of the code. In that case, Q/A should be testing the a build with Assertions disabled.

To date, we have always had Q/A operating on our Relesae builds, as that was the code that would ship. However, I am thinking about trying a pattern where our really early releases to Q/A would go out with assertions enabled. Then as we get closer to shipping, we will notify them that their builds have assertions disabled.

What do you guys think?

Answer 1

We ship release builds with debug symbols, so that the performance is right (extensive use of debug output and assertions can slow things down a little) but they still report meaningful stack traces if an exception occurs.

For exceptions, we have the general rule only to catch exceptions which we know how to handle, so that they pop up in QA if something has not been thought of. Catch-all are prohibited in general in our company.

Answer 2

We release both to Q/A, and test passes complete on both. If you are heavy on automated testing, this just becomes an issue of having extra hardware to run the tests.

Release versions must be tested, since they are what actual customers use. Debug versions contain addition asserts/validations/tracing/etc., which are incredibly useful at finding non-obvious bugs.

Answer 3

Using debug builds for QA in early development stages and switching to release builds later is exactly what we are also doing.

Answer 4

I would definitely advocate to have QA review not only the releases that go to the customer, but also some intermediate builds, maybe every two weeks or once per month.

This is in accordance with the principle to check your product as early in the development process as possible. If you are only doing it when you are releasing a version, you are doing it too late.

And I would give them the debug build, with asserts enabled for those early tests. If you have an assert in the code that fails, you have an error. Either the code is wrong, or the assert. Either way, you want QA to test for that.

Answer 5

Me too on Debug early/Release late; my previous employer's official policy (sometimes breached, but not by me) was to test Debug builds until Beta, and Release builds thereafter. It's obviously worth running Debug builds occasionally until you ship, but an unfortunate political reality is that a lot of program managers won't have a bug fixed if it's reported against a Debug build.