How do you tailor your dev process around adding unit tests to the build?

Question

Hi folks,

I have ran into a situation at work where it has been proposed to add our units tests to the Cruise Control continuous build process. The units test would be added as another project in CC, so that we would have a "Normal Build" project and a "Build and Test" project. We currently use the "Normal Build" process to a) make sure that no developers have broken the build, and b) make sure that the build is not broken when a release is about to be pushed to QA/Production.

By adding unit tests to the process we can a) be notified when a unit test has been broken so we will check to make sure that it was not one that we wrote, and b) ensure that no units are failing before a build gets released to QA/Production. So I understand the benefits, but I still am concerned about how this will affect our development process. Here's the scenario's that I see occurring...

Current process...

• Build Server builds Cruise Control "Normal Build" project
• Light goes red (in cruise control) when build fails
• Blamed developer gets a troll

The new process...

• Another CC project is added for "Build and Test" (now two projects)
• Light is green for "Normal Build" project
• Light goes red for "Build and Test" project
• Blamed developer gets...

Scenario one: Blamed developer get's NOTHING. We're doing TDD and it's OK if unit tests fail. "Build and Test" project is red and we're not about to push a build to QA.

Scenario two: Blamed developer gets TROLL, "Build and Test" project should never be red.

Anyway, I'm curious to read what the thoughts are on this process? Please keep them brief. Does your team includes units tests in the build. If so, do you follow TDD? Do you always strive to keep your builds with tests as all passing?

Thanks!

Answer 1

build should run the tests, always passing, and yes for TDD

also, if you have automated functional testing, that should be run also. With functional testing you may not have all the tests passing, however you should never start failing a test that was passing.

(ie.... you can have a bunch of functional tests that will demonstrate that current stuff being worked on is doing the right thing, but until its completed it won't pass. )

Answer 2

Anyway, I'm curious to read what the thoughts are on this process? Please keep them brief. Does your team includes units tests in the build. If so, do you follow TDD? Do you always strive to keep your builds with tests as all passing?

Well, we're not always perfectly consisten about it, but in principle: yes, and yes.

Point 1:

IMHO, you really must run your tests in the daily build. Otherwise test failures in places where you didn't expect them will go unnoticed for a while - this defeats the whole purpose of unit testing. It's only useful if test failures are noticed immediately.

Obviously, during development there will be phases where you break tests (e.g. refactoring). But these should be as short as possible (minutes, ideally), and you typically only check in once they pass (just as you only check in once stuff compiles). If you need to do more extensive rework, you need a private branch :-).

Point 2:

As to what happens when tests break: Well, you fix them (you, or whoever broke them). Strict reprimands should be unnecessary in a good team. But that is really a social question...

Answer 3

Whether or not you use TDD, don't checkin unless/until it passes its tests. Or if you like/must, you can checkin broken code/tests to a 'private' branch whose testing isn't asserted.