Remembering to run tests before commit

Question

We have a decent set of unit tests on our code and those unit tests run in under 2 minutes. We also use TeamCity to do a build and to run tests after each check in. However, we still get issues where a developer "forgets" to run all the tests before a commit resulting in TeamCity failure which if this check in was done at 6PM may stay broken over night.

"Forgets" is a generic term, there a couple other common reasons why even remembering to run the tests could result in TeamCity failure. Such as.

-> A developer only checks in some of the modified files in his/her workspace.
-> A file was modified outside of eclipse such that eclipse's team synchronize perspective does not detect it as dirty.

How do you deal with this in your organization?

We are thinking of introducing "check in procedure" for developers which will be an automated tool that will automatically run all unit tests and then commit all of the "dirty" files in your workspace. Have you had any experience with such process? Are you aware of any tools which may facilitate this process? Our dev environment is Python using Eclipse's PyDev plugin.

Answer 1

For mercurial you can use hooks, that will run tests and only allow push on success. But this may require a lot of time for a push (but developer has to run those tests anyway).

Or you can just have own set of bash scripts, that will run test and only then run commit command. For example for django and svn commit it could look as simple as this:

./manage.py test && svn commit $@

Or there is another way: if anyone commits code, that doesn't pass test, he pays some sum. Soon people will remember to test, for they won't like the notion of paying money ;-)

Answer 2

I think it is more of a social problem rather than a deficiency of the automated systems.

Yes, you can improve the systems in place, but they will be no match for someone thinking of the implications of their commit, and testing it before they hit commit.

Answer 3

In one of the teams I was working before we had an agreement that anyone who breaks the tests buys bacon sandwiches for the whole team the next morning. Its extreme, but it works perfectly!

Answer 4

TeamCity has some support for pretested commit; if your development team is using a supported IDE, you might look into that.

In my company, we don't worry about it too much - our pattern looks something like this.

(a) each developer has their own project configuration in TeamCity, with roots pointing to their own sandbox. They are allowed to do anything they like here.

(b) the development team has an integration sandbox, where all changes are delivered. A project encapsulates the configurations which monitor this branch in the source control system. Dev Leads get to make up the rules here, but that rule is almost always "it must stay green". I'd have to look at the exact percentage of clean builds - it's not a perfect record, but its high enough that I've never been tempted to insist that the developers be more disciplined about running tests.

(c) the actual deliver comes from a Main stream, which Shall Stay Green (tm). Dev lead is responsible for delivering a clean snapshot of integration to the main stream on a well defined schedule. This project is the one that actually generates the installers that are delivered to testing, the bits that go into escrow, etc.

One reason that you might get a way with a more aggressive policy than we do is that our build cycle is slow - on the order of four hours. If we were an order of magnitude smaller, with a poor success rate, I might argue a different case.