How important is fixing the nightly build promptly?

Question

We have an automated build server that builds our code nightly, which is useful for us since not everyone on our team can build the entire source tree. Lately, some members of the team are becoming more lax about fixing build errors promptly; sometimes weeks will go by without a successful build. I even overheard one developer say, "the build is already broken, now is a good time to add [some breaking change]." Since I work on the the code the furthest downstream, I am usually working with parts of the tree that are woefully out of sync with the source code repository, which makes it very difficult to test changes before I submit them.

I feel like we're losing most of the benefit of having a nightly build since it is continually broken. Am I way off base here, or should fixing the build be a higher priority?

Answer 1

Those devs clearly need to be kicked back into shape.

I'd suggest building at least a few times daily, if not upon checkins. And once you got a successful build cycle going again, have a go (in a joking way) at the person who broke the build - when it happens.

Everyone needs to take ownership of the codebase and take responsibility.

To be honest, it also is about having some pride in your craft. If ultimately people don't give a damn if the build is broken, and they don't after being asked to sort it out, it sounds they'd be better off doing some other job.

Answer 2

Fixing the nightly build should be the highest priority. As you said, if they are broken, they have no value. If people wish to check in code that causes breakage, they should do this on a branch and only merge it in when it is tested.

Answer 3

The longer you put off fixing it, the longer it will take to fix.
If it's fixed immediately, the things that cause it to be broken should be fresher in everyone's head. Breaking changes could also be piling up making it that much more of a headache to fix later.

Answer 4

It's critical to get it fixed. The longer you put it off, the more things you're going to find later. How can someone tell if their changes have broken the build, if they don't start with a clean build?

Our standard is to have all our unit and functional tests run "green" on a neutral integration box after a commit. Of course, test-driven development is appropriate to our situation, but may not fit yours. If you're not even able to build the project, there are probably bad surprises lurking in previous commits.

If it's so big that the time it takes to build it is standing in the way of getting it fixed, techniques like breaking it up into smaller projects and continuous integration may help.

Answer 5

A friend of mine told me about his team that had the Zucchini of Doom. Anyone breaking the nightly build had to display the ZoD on their desk. This vegetable was in a fairly advanced state of decomposition, which sent out the message quite clearly that a broken build was not something to be tolerated.

If the team isn't motivated enough to keep the nightlies building then this is something that should be enforced/encouraged by the managers.