Punishment for breaking the build

Question

I'm sitting here frustrated that a team member broke the build and then goes on a long-weekend vacation. What happens on other teams when someone breaks the build? What would you consider a fair punishment?

Answer 1

You seem to practice agile, you better check it out and fix it, NOW!!! :)

Update: Punishment for the guy who broke the build, fix the next broken build ;)

Answer 2

We have a rubber chicken that hangs outside the cubicle of the person who most recently broke the build.

Answer 3

We all do it from time to time, but some people are definitely worse than others.

A publicly visible count of build breaks with blame attached ought to do it. The common culprits will become obvious.

Do it in a light hearted way on one of your common servers.

Answer 4

The person breaking the build becomes the build administrator

Answer 5

One placed I worked, we had a broom. Its use came from checking in "sweeping changes" which usually broke something or other. So we appropriated a broom and it eventually changed into an I-broke-the-build indicator.

Answer 6

I "suggest" donations to charity organisations. Works great for people coming late for meetings, too.

Answer 7

The person who broke the build is now responsible for testing the build.

(with the benefit that that programmer will be incentivized to catch the next offender)

Answer 8

  And the lord said unto man

"He who checks in files which
   break the build for others
     shall pay a penance of
              doughnuts"

            That is the law

Answer 9

this is more a training issue, punish the team lead I say

Answer 10

I know one team that made the developer who broke the build wear a dress for a day.

Answer 11

"He whose release doth cause offence shall bake a cake in recompense"

Serves us very well, and we get some great cakes out of it! Helps people take responsibility but without feeling beat up about it.

Answer 12

About five years ago things got so bad that I got very "librarian" about it, and started a spreadsheet with WALL OF SHAME in large letters at the top and entered every breakage into it. Each row had the name of the person responsible, the reason why it occurred, and an explanation of the steps taken to ensure it didn't happen again. It was surprisingly effective, but excruciating to have to administer. And people complained about it. So I told them "This is an optional scheme - if you don't want to take part, just never break the build."

Now we have continuous integration, the full nightly build only breaks extremely rarely - if there's a powercut or a disk goes pop. Though the old timers still speak in hushed tones of the dark days when we had THE WALL.

Answer 13

Is death a bit harsh? :)

Answer 14

we use a Mr. Potato Head. When the build is broken then the Potato Head gets set on the edge of the breaker's cube for all to see.

Answer 15

I guess someone could have to write "I will not break the build" 1000 times, but then again, programmers would just do:

for (int i=0; i<1000; i++) System.Console.WriteLine("I will not break the build");

but in the projects I've use continuous integration on, we used CCTray, and it popped up with a message to all developers on the project within a minute of checking in broken code. All the programmers would just convene at the desk of the culprit shaking their fists and suggesting that it be rolled back or fixed code be submitted. It usually wasn't an issue.

Answer 16

My experience with build breaks has been that the guys checking in the most breaks are often the ones doing the most work.

Yes, there are sloppy developers checking in crap. There are also some geniuses that break on one platform or another. When the average developer breaks the build in 1% of their check-ins, and the geniuses break the build in 0.1% of their checkins, the geniuses still break the build more.

Sometimes, you need to know why. Is there a cultural or process reason? I know that in our case, those "sweeping changes" alluded to above were part of the problem. If you need to touch 120 files once in a while (yes, I've had a delivery in clearcase within the last year with ~120 elements in it, and, yes, it almost broke the build, but I managed to catch the files I missed before the next build - this time), you need to either expect build breaks or find a procedural improvement that can catch them faster/easier (continuous integration, for example, probably would have been sufficient). Ostracising your star developers who are the only ones gutsy enough to make sweeping changes probably isn't the right answer.

Answer 17

The best way how to deal with this is to create automated checkin policy, which prevents committing (Checkin in) broken builds, or at least reports the broken build very soon after checked in.

One possible implementation for this can be:

• have a dedicated build server
• have a set of automated functionality tests (executed on the build server) which check the build can successfully built, can be run and performs basic functionality OK

Answer 18

$1 in a bucket that is saved for release celebration, beers, etc...

Answer 19

We have a whiteboard on the wall in the hallway. For any break that should have reasonably been detected, if your breakage makes it past the 5-10 min build and on to the 2 hour build, your name gets added to the board.

We also use the continuous integration game with Hudson, so people want to be the one with the highest score.

Answer 20

Yes, breaking the build and going on vacation (or even home for the night) is bad. I'm talking about the occasional break that gets fixed right away, which is what most of the replies in this thread seem to be referring to.

On our team we have no such "punishment policy" no matter how unwritten or informal. I see no need, because we all know that breaking the build keeps the others on the team from updating or checking in (and that's all it does). That's plenty of motivation for us to be careful.

So where is the need for punishment? To me it smells of 8 year-olds on a playground pointing at each other singing: "nyaa, nyaa, nyaa-nyaa, nyaa... you broke the bui--ild."

Answer 21

None. Encourage programmers not to be scared. It is when they break but not fix the build, then may {insert choice higher being} have mercy on their soul.

Answer 22

A guy I worked with, in a past job, worked in a place where if you broke the build, you got to have, all to yourself, in your very own cubicle:

A giant cardboard cutout of Jar-Jar Binks.

Answer 23

It's my understanding that even "fun" broke-the-build markers are considered to be less than optimal because they discourage frequent check-ins, which prevents problems from being caught earlier while they're smaller, not to mention increasing the difficulties of merging.

On the other hand, I did like the dollar into the beer fund suggestion.

Answer 24

We have a CI system that will send an email and let people know that the build has broken (tests are failing or whatever). It saves us a lot of time. But seriously...

It's just not a big deal.

Why? We use git, so one of two things happens:

1. The rest of us continue to work without those newest changes until the guy who pushed bad code fixes his code.
2. One of us throws away his changes and resets the main branch and lets him rework it and try it again.

The biggest problem I've run into is having people who don't check in changes. I'd rather have pushed changes and build an after-the-fact branch to stash stuff away while it gets fixed than to have people who sit on changes for months to hide stuff.

Answer 25

We have just started throwing shoes at the offender.

Answer 26

Have the build server automatically start playing country western music over a loud speaker which cannot be shut off until the build is fixed...

Answer 27

I did hear of one place that had a big green lava lamp, and a big red lava lamp. When the build was okay, the green one was lit. When the build broke, the red one would light up.

Not really a punishment, but a pretty cool I thought.

Answer 28

We have a big red color foam hand pointing index finger. Whoever breaks a build or does something nasty that hand is kept at his/her desk with finger pointing towards them.

Answer 29

Revert the offending commit.

Having to merge changes when they return is usually punishment enough.

Answer 30

A small, stuffed Cthulhu in a coffee cup that sits on the culprit's desk, hight "Works". The developers always say "Works on my machine", so we made it so!

Answer 31

spank them with a mechanical corset

Answer 32

No punishment is severe enough!

Answer 33

Hi!

I'm strongly disagree with any punishment. A simple friendly and private note from the others is enough.

Punishing others could damage moral. Maybe it sound fun, but could hurt someone feeling and to avoid mistakes in the future it minimize his development efforts (loosing creativity).

For me CI means to realize mistakes as soon as possible and not, developing without mistakes. Every human makes mistake. Any kind of solution, which based on: "that this is so simple, no one ever going to mess up" is going to fail.

As always if the build can be broken than it is architectural mistake. Saying don't break the build is no different as the note: Please use large capitals in this text box or we cannot parse your order and can seriously damage our servers. Or with other words. How do you going to develop user friendly stable applications, if the tools and processes what you use and develop for your self aren't satisfying these criteria.

If you use any kind of script or build tool you can create a process, where the check-in procedure involves code update, build, QA and check-in.

However this could only lower the number of broken builds. As for example special file extensions wasn't checked in into the last step could affect the build.

The solution is double QA. When you fix someone bike, you first test it then the customer test it as well before accepting it.

If you use transactional SCM, than you can maintain two branch. One is where everyone checks in the code. On that branch a build tool runs, which builds and QA against every transaction (check ins). If it succeed it pass the transaction to the main branch. If not it throws away that transaction. (Also notify the checker :-) ).

With this the developers will have a branch, where they always have stable code. And a branch where they can check-in without risk.

But have in mind the effort and benefits. If you are working locally with 2 - 5 person, spending to much time creating this environment maybe not worth the effort. A simple please fix it could be enough. On larger teams working in different location could be handy.

Also this won't answer code QA issues on some level, recognize integration issues on early level and point out invalid business logic.

Answer 34

At my current company I had an svn precommit hook that would build the project with the new changes on the build server before it would allow the commit to be accepted. All was well until the project grew and build started to take longer and handling special cases of commits happening while a build was in progress due to a prior commit by someone else. While it was in place it was nice though because no build breaks ever occurred and it gave us a place to run any future automated tests or code analysis as well in order to also reject the commit on that basis. At the end the developer would get a little message back from SVN saying commit accepted or commit rejected and the reason.

Now since I've disabled this due to the slow performance of the build we just tend to declare the usual shenanigans and rally the villagers until we get an apology.