views:

5987

answers:

40

I'm sitting here frustrated that a team member broke the build and then goes on a long-weekend vacation. What happens on other teams when someone breaks the build? What would you consider a fair punishment?

+4  A: 

You seem to practice agile, you better check it out and fix it, NOW!!! :)

Update: Punishment for the guy who broke the build, fix the next broken build ;)

Vyas Bharghava
+10  A: 

We have a rubber chicken that hangs outside the cubicle of the person who most recently broke the build.

jalbert
+49  A: 
Greg Beech
Hehe, I'd like to see that outfit...
Andreas Magnusson
I'll have to see if I can dig out a photo of me the last time I broke the build (did I mention we take photos too...?)
Greg Beech
This is just getting better and better!
Andreas Magnusson
I say we all vote this down until we get a picture!
Aardvark
I don't think embarrassing people is a good strategy.
David G
One problem with trying to embarrass people is that not everyone would be embarrassed by that. The last thing you want is someone breaking the build so he can pick up pizza dressed as a bee.
Dave DuPlantis
It's not really that embarrassing in our environment (it might be in others but we're very informal and casual). It's just intended to be a bit of fun, which is how everyone takes it.
Greg Beech
Hehe, cool. Where can we get one?
Andreas Magnusson
To protect the innocent? But he's guilty!
Germán
I see no picture or link?
Oorang
A: 

We all do it from time to time, but some people are definitely worse than others.

A publicly visible count of build breaks with blame attached ought to do it. The common culprits will become obvious.

Do it in a light hearted way on one of your common servers.

Simon
+9  A: 

The person breaking the build becomes the build administrator

harriyott
What? You're not using automated builds?
Kelly French
+22  A: 
thatismatt
If I had such thing on my desk, I couldn't wait for a broken build to use it. +1 for the suggestion though.
OregonGhost
could could probably code something watch for broken builds and aim and fire it at the person who broke it automatically.
Scott Cowan
I got one that didn't work as well as described on the web site.
stephenbayer
Yes; scott, that's exactly what I wanted to implement here. We didn't get around to it though. Definitely the most awesome solution.
Noon Silk
+1  A: 

One placed I worked, we had a broom. Its use came from checking in "sweeping changes" which usually broke something or other. So we appropriated a broom and it eventually changed into an I-broke-the-build indicator.

Greg Hewgill
+8  A: 

I "suggest" donations to charity organisations. Works great for people coming late for meetings, too.

Nicolai Reuschling
Shouldn't it be, I suggest "donations"? I find the quotations around the word suggest to be alittle odd, and maybe a little... insidious? It could just be too early in the morning however.
James McMahon
I guess it is 'suggest' spelled 'o-r-d-e-r'.
Remou
Well, I am not a native speaker. Please be gentle with my wording here.
Nicolai Reuschling
He makes donation offers they can't refuse ;-)
ypnos
I would be more hesitant to be late for meetings if the punishment was *removing* money from a pot which would otherwise be going to charity. For instance, if there was a budget for $1000 to go to charity out of the company's funds - and $5 removed from it if you're late.
Smashery
+1 for the charity for coming late to meetings ... You have Outlook, it reminds you TWICE ... come on time!
Martin
+5  A: 

The person who broke the build is now responsible for testing the build.

(with the benefit that that programmer will be incentivized to catch the next offender)

John Mulder
+132  A: 

  And the lord said unto man

"He who checks in files which
   break the build for others
     shall pay a penance of
              doughnuts"

            That is the law

Jeff Atwood
MY DAILY REP-WHORING
Jeff Atwood
This is also the penance at here, although chocolate is an acceptable substitute.
Pev
Mmm, I like donuts. And chocolate.
Andreas Magnusson
How does that work over skype?
Greg Hewgill
Yea, and the doughnuts were brought, and they were consumed, and the Lead saw this, and it was good. And the Lead took a senior programmer aside, and said, "Let this N00b break the build again tomorrow, for doughnuts are pleasing to Me."
Dave DuPlantis
Image blocked by overzealous company proxy server
Alex Angas
Best. Response. Ever.
Dan Esparza
Man, I though the donuts originated in my software team. Dang. I tell you it works.
Daniel Paull
@Dave DuPlantis - that is the best comment I've ever seen on SO.
dotjoe
+43  A: 
keparo
Hate check-in reverters :) Give me a moment to fix it, will you!It's a different thing if I'm away, but if you just need to get something to build while I'm fixing it then for pete's sake just leave your change in your workspace.
romkyns
+1 for "aim for the face"
iandisme
A: 

this is more a training issue, punish the team lead I say

redsquare
Then Sun Tzu said, “If the instructions and words of command are not clear and distinct, if orders are not thoroughly understood, the general is to blame. But if the commands are clear and the soldiers disobey, then it is the fault of the officers.”
Andreas Magnusson
He [Sun Tzu] immediately ordered the women who were at the head of the two troops to be beheaded.
Andreas Magnusson
+4  A: 

I know one team that made the developer who broke the build wear a dress for a day.

bmatthews68
I know developers that wear dresses on regular days... do you have any women at your company?
emddudley
Actually on that project which was split over 3 sites (2 US, 1 UK) there was not one woman working as a developer.
bmatthews68
+3  A: 

"He whose release doth cause offence shall bake a cake in recompense"

Serves us very well, and we get some great cakes out of it! Helps people take responsibility but without feeling beat up about it.

andygeers
Problem is of course whether the person's cake-baking skills is on pair with their build-breaking skills...
Andreas Magnusson
So change "bake" to "buy" ...
Dave DuPlantis
"Four large eggs. One cup semi-sweet chocolate chips. 3/4 cups butter or margarine. Fish shaped crackers. Fish shaped solid waste. Alpha resins. Unsaturated polyester resin. 2/3 cups granulated rhubarb. One tablespoon all-purpose rhubarb. One cross borehole electro-magnetic imaging rhubarb." -GLaDOS
Erik Forbes
Be careful of disgruntled workers -- who might spike the cake with say.. x-lax or other more natural ways of punishing people!
torial
+7  A: 

About five years ago things got so bad that I got very "librarian" about it, and started a spreadsheet with WALL OF SHAME in large letters at the top and entered every breakage into it. Each row had the name of the person responsible, the reason why it occurred, and an explanation of the steps taken to ensure it didn't happen again. It was surprisingly effective, but excruciating to have to administer. And people complained about it. So I told them "This is an optional scheme - if you don't want to take part, just never break the build."

Now we have continuous integration, the full nightly build only breaks extremely rarely - if there's a powercut or a disk goes pop. Though the old timers still speak in hushed tones of the dark days when we had THE WALL.

Daniel Earwicker
+13  A: 
philippe
+4  A: 
philippe
for (int i = 0; i < 1000; i++) Console.WriteLine("I will not break the build.");
Martin
+28  A: 
Penguinix
I am laughing so hard right now ;-)
matt_dev
+1 for old woman giving the finger
Martin
+13  A: 

Is death a bit harsh? :)

Greg Whitfield
depends on the proximity to the launch date.
Rob Allen
we should be able to give +1 to comments lol great one
fmsf
+1 for Rob's comment.
Bill Williams
A: 

we use a Mr. Potato Head. When the build is broken then the Potato Head gets set on the edge of the breaker's cube for all to see.

codeLes
+2  A: 

I guess someone could have to write "I will not break the build" 1000 times, but then again, programmers would just do:

for (int i=0; i<1000; i++) System.Console.WriteLine("I will not break the build");

but in the projects I've use continuous integration on, we used CCTray, and it popped up with a message to all developers on the project within a minute of checking in broken code. All the programmers would just convene at the desk of the culprit shaking their fists and suggesting that it be rolled back or fixed code be submitted. It usually wasn't an issue.

stephenbayer
+93  A: 

My experience with build breaks has been that the guys checking in the most breaks are often the ones doing the most work.

Yes, there are sloppy developers checking in crap. There are also some geniuses that break on one platform or another. When the average developer breaks the build in 1% of their check-ins, and the geniuses break the build in 0.1% of their checkins, the geniuses still break the build more.

Sometimes, you need to know why. Is there a cultural or process reason? I know that in our case, those "sweeping changes" alluded to above were part of the problem. If you need to touch 120 files once in a while (yes, I've had a delivery in clearcase within the last year with ~120 elements in it, and, yes, it almost broke the build, but I managed to catch the files I missed before the next build - this time), you need to either expect build breaks or find a procedural improvement that can catch them faster/easier (continuous integration, for example, probably would have been sufficient). Ostracising your star developers who are the only ones gutsy enough to make sweeping changes probably isn't the right answer.

Tanktalus
I love this answer!
Wayne Koorts
I agree with this. When I started my development career, I assumed that breaking the build was a cardinal sin. But everybody does it now and then, me included. It's not the end of the world. In many cases you can see how to fix it yourself when you discover that someone else broke it. If not, go talk to that person and find out how to fix it. There's no point in getting adversarial over something that happens every so often. I'm working in a place where one guy is a real stickler for this, and all it does is create workplace tension for no good reason.
Kyralessa
+3  A: 

The best way how to deal with this is to create automated checkin policy, which prevents committing (Checkin in) broken builds, or at least reports the broken build very soon after checked in.

One possible implementation for this can be:

  • have a dedicated build server
  • have a set of automated functionality tests (executed on the build server) which check the build can successfully built, can be run and performs basic functionality OK
Suma
+25  A: 

$1 in a bucket that is saved for release celebration, beers, etc...

kenny
I like it, contribute to group morale :)
JettGeek
That's what we used on the Blacksite project.
280Z28
A: 

We have a whiteboard on the wall in the hallway. For any break that should have reasonably been detected, if your breakage makes it past the 5-10 min build and on to the 2 hour build, your name gets added to the board.

We also use the continuous integration game with Hudson, so people want to be the one with the highest score.

Joshua McKinnon
+14  A: 

Yes, breaking the build and going on vacation (or even home for the night) is bad. I'm talking about the occasional break that gets fixed right away, which is what most of the replies in this thread seem to be referring to.

On our team we have no such "punishment policy" no matter how unwritten or informal. I see no need, because we all know that breaking the build keeps the others on the team from updating or checking in (and that's all it does). That's plenty of motivation for us to be careful.

So where is the need for punishment? To me it smells of 8 year-olds on a playground pointing at each other singing: "nyaa, nyaa, nyaa-nyaa, nyaa... you broke the bui--ild."

Tim Tonnesen
I agree. unprofessional.
Tim
I once worked on a team where one developer _always_ broke the build. EVERY check in. And he was a senior developer. Because there was no policy, there was no discipline. And discipline was most certainly needed.
salt.racer
I think while I agree about it potentially being unprofessional, in a good team the 'punishments' are known to be there to raise morale rather than point and laugh. Everyone will break the build at some point, so it's not personal. If someone gets upset about it just back off and don't 'punish' them next time.
Kurucu
+24  A: 

None. Encourage programmers not to be scared. It is when they break but not fix the build, then may {insert choice higher being} have mercy on their soul.

vfilby
I worked on an enormous project that took nearly 24 hours to build, so breaking it meant a lot of time lost. 2000 people worked on that project, and many were waiting for a good build to come out, so that made it all the more expensive. Don't break the build.
Jay Bazuzi
Programmers scared of breaking builds will be scared of making changes, and therefore scared of working on the code. Not productive in the least. Revisit program without fear. Sounds like the project was a little too monolithic and poorly architected. 2000 people shouldn't be responsible for or dependent on a single build. Individual components with stable version would have been a better choice.Alternatively implement some measures to detect build break changes before they get checked into source control.
vfilby
imho, continuous integration that notifies everyone of build-breaking makes breaking the build _no big deal at all_ because you'll fix it. You lose like 2 seconds of time. So CI makes punishment obsolete in my experience.
apollodude217
+24  A: 

A guy I worked with, in a past job, worked in a place where if you broke the build, you got to have, all to yourself, in your very own cubicle:

A giant cardboard cutout of Jar-Jar Binks.

Kyralessa
Oh, the horror!!
Evan
+7  A: 

It's my understanding that even "fun" broke-the-build markers are considered to be less than optimal because they discourage frequent check-ins, which prevents problems from being caught earlier while they're smaller, not to mention increasing the difficulties of merging.

On the other hand, I did like the dollar into the beer fund suggestion.

J.T. Hurley
+20  A: 

We have a CI system that will send an email and let people know that the build has broken (tests are failing or whatever). It saves us a lot of time. But seriously...

It's just not a big deal.

Why? We use git, so one of two things happens:

  1. The rest of us continue to work without those newest changes until the guy who pushed bad code fixes his code.
  2. One of us throws away his changes and resets the main branch and lets him rework it and try it again.

The biggest problem I've run into is having people who don't check in changes. I'd rather have pushed changes and build an after-the-fact branch to stash stuff away while it gets fixed than to have people who sit on changes for months to hide stuff.

Dustin
+7  A: 

We have just started throwing shoes at the offender.

Martin Brown
ROFLMFAO its good even a week later.
Unkwntech
Good one. Given the fact throwing shoes started again .. :)
Guru
+9  A: 

Have the build server automatically start playing country western music over a loud speaker which cannot be shut off until the build is fixed...

Evan
Collective punishment! :) This reminds me of tales I heard from Romania, where if a few people didn't pay their hot water bill, they'd shut off hot water for the whole apartment building.
Kyralessa
+1  A: 

I did hear of one place that had a big green lava lamp, and a big red lava lamp. When the build was okay, the green one was lit. When the build broke, the red one would light up.

Not really a punishment, but a pretty cool I thought.

Evan
A: 

We have a big red color foam hand pointing index finger. Whoever breaks a build or does something nasty that hand is kept at his/her desk with finger pointing towards them.

Bhushan
+11  A: 

Revert the offending commit.

Having to merge changes when they return is usually punishment enough.

Jeffrey Fredrick
I like it! It's punishment, but also doesn't upset the more politically correct people :)
Kurucu
+2  A: 

A small, stuffed Cthulhu in a coffee cup that sits on the culprit's desk, hight "Works". The developers always say "Works on my machine", so we made it so!

A: 

spank them with a mechanical corset

banister
A: 

No punishment is severe enough!

fastcodejava
+1  A: 

Hi!

I'm strongly disagree with any punishment. A simple friendly and private note from the others is enough.

Punishing others could damage moral. Maybe it sound fun, but could hurt someone feeling and to avoid mistakes in the future it minimize his development efforts (loosing creativity).

For me CI means to realize mistakes as soon as possible and not, developing without mistakes. Every human makes mistake. Any kind of solution, which based on: "that this is so simple, no one ever going to mess up" is going to fail.

As always if the build can be broken than it is architectural mistake. Saying don't break the build is no different as the note: Please use large capitals in this text box or we cannot parse your order and can seriously damage our servers. Or with other words. How do you going to develop user friendly stable applications, if the tools and processes what you use and develop for your self aren't satisfying these criteria.

If you use any kind of script or build tool you can create a process, where the check-in procedure involves code update, build, QA and check-in.

However this could only lower the number of broken builds. As for example special file extensions wasn't checked in into the last step could affect the build.

The solution is double QA. When you fix someone bike, you first test it then the customer test it as well before accepting it.

If you use transactional SCM, than you can maintain two branch. One is where everyone checks in the code. On that branch a build tool runs, which builds and QA against every transaction (check ins). If it succeed it pass the transaction to the main branch. If not it throws away that transaction. (Also notify the checker :-) ).

With this the developers will have a branch, where they always have stable code. And a branch where they can check-in without risk.

But have in mind the effort and benefits. If you are working locally with 2 - 5 person, spending to much time creating this environment maybe not worth the effort. A simple please fix it could be enough. On larger teams working in different location could be handy.

Also this won't answer code QA issues on some level, recognize integration issues on early level and point out invalid business logic.

David
First off, as I said in a comment to the original question it was all done in jest between equals. I'm not his manager. The key point was not breaking the build but taking vacation after doing so. This is a highly skilled dev who should know better we're talking about. Secondly, in work life you may not always get to choose the tools nor the way code gets developed. If you're working on a 10+ yrs old code base, chances are you're stuck with a great deal of old tools and practices.
Andreas Magnusson
Everyone makes mistake. But with punishment we are only damage the moral, but the problem will exists.I know what you are talking about. For e.g.: currently the whole tech team went for vacation after bringing the new system alive. But before that they left me a note that please undo the build fix what I introduced because they not planed it. Not because it is not working. It is not yet decided.Now these guys is the not now and not ever types. They don't have the skill to fix or undo this. So it will stay there and they soon find something else to manage. Sometimes pro-activity is the key
David
A: 

At my current company I had an svn precommit hook that would build the project with the new changes on the build server before it would allow the commit to be accepted. All was well until the project grew and build started to take longer and handling special cases of commits happening while a build was in progress due to a prior commit by someone else. While it was in place it was nice though because no build breaks ever occurred and it gave us a place to run any future automated tests or code analysis as well in order to also reject the commit on that basis. At the end the developer would get a little message back from SVN saying commit accepted or commit rejected and the reason.

Now since I've disabled this due to the slow performance of the build we just tend to declare the usual shenanigans and rally the villagers until we get an apology.

jpierson