Should failing tests make the continuous build fail?

Question

If one has a project that has tests that are executed as part of the build procedure on a build machine, if a set tests fail, should the entire build fail?
What are the things one should consider when answering that question? Does it matter which tests are failing?

---

Background information that prompted this question:

Currently I am working on a project that has NUnit tests that are done as part of the build procedure and are executed on our cruise control .net build machine.

The project used to be setup so that if any tests fail, the build fails. The reasoning being if the tests fail, that means the product is not working/not complete/it is a failure of a project, and hence the build should fail.

We have added some tests that although they fail, they are not crucial to the project (see below for more details). So if those tests fail, the project is not a complete failure, and we would still want it to build.

One of the tests that passes verifies that incorrect arguments result in an exception, but the test does not pass is the one that checks that all the allowed arguments do not result in an exception. So the class rejects all invalid cases, but also some valid ones. This is not a problem for the project, since the rejected valid arguments are fringe cases, on which the application will not rely.

Answer 1

If a unit test fails, some code is not behaving as expected. So the code shouldn't be released.

Although you can make the build for testing/bugfixing purposes.

Answer 2

If it's in any way doable, then do it. It greatly reduces the broken-window-problem:

In a system with no (visible) flaws, introducing a small flaw is usually seen as a very bad idea. So if you've got a project with a green status (no unit test fails) and you introduce the first failing test, then you (and/or your peers) will be motivated to fix the problem.

If, on the other side, there are known-failing tests, then adding just another broken test is seen as keeping the status-quo.

Therefore you should always strive to keep all tests running (and not just "most of them"). And treating every single failing test as reason for failing the build goes a long way towards that goal.

Answer 3

If you felt that a case was important enough to write a test for, then if that test is failing, the software is failing. Based on that alone, yes, it should consider the build a failure and not continue. If you don't use that reasoning, then who decides what tests are not important? Where is the line between "if this fails it's ok, but if that fails it's not"? Failure is failure.

Answer 4

I think a nice setup like yours should always build successfully, including all unit tests passed.

Like Gamecat said, the build itself is succeeded, but this code should never go to production.

Imagine one of your team members introducing a bug in the code which that one unit test (which always fails) covers. It won't be discovered by the test since you allow that one test to always fail.

In our team we have a simple policy: if all tests don't pass, we don't go to production with the code. This is also a very simple to understand for our project manager.

Answer 5

In my opinion it really depends on your Unit Tests,... if your Unit tests are really UNIT tests (like they should be => "reference to endless books ;)" ) then the build should fail, because something is not behaving as should...

but most often (unfortunately to often seen), in so many projects these unit tests only cover some 'edges' and/or are integration tests, then the build should go on (yes, this is a subjective answer ;)

in short: do you know the unit tests to be fine: fail; else: build on

Answer 6

The real problem is with your failing tests. You should not have a unit test where it's OK to fail because it's an edge case. Decide whether the edge case is important or not - if not then delete the unit test; if yes then fix the code.

Like some of the other answers implied, it's definitely a code smell when unit tests fail. If you live with the smell, then you're less likely to spot the next problem

Answer 7

All the answers have been great, here is what I decided to do:

Make the tests (or if need be split a failing test) that are not crucial be ignored by NUnit (I remembered this feature after asking the question). This allows:

• The build can fail if any tests fail, hence reducing the smelliness
• The tests that are ignored have to be defended to project manager (whomever is in charge)
• Any tests that are ignored are marked in a special way

I think that is the best compromise, forcing people to fix the tests, but not necessarily right away (but they have to defend their decision of not fixing it now since everyone knows what they did).

---

What I actually did: fixed the broken tests.