How do you manage yourself after discovering you are responsible for a critical bug?

Question

You've discovered that you are the author of a critical bug that has made it into customer's hands and it's caused problems with their operations.

You have a solution that should fix the problem, but you are still responsible for the problem in the first place.

You've disclosed that you are the author of the bug, and have worked with customers to mitigate the problem.

We're all human, and capable of making mistakes, but how do you deal with this failure?

Answer 1

Honesty, is the best way. Admit that there was a bug, and admit the cause. You do not need to tell customers it was you personally responsible for the bug but the company as a whole.

When you communicate internally that might be different.

The other thing is to be apologetic. I work for a small startup, and a couple of our users have my cell phone, when they have an issue they'll call me up, and you can tell they are very frustrated. First I apologize for the pain they are experiencing, secondly I work with them to find a resolution as quickly as possible either via a work around or by modifying and redeploying our site.

These users despite the pain I caused them, still love the product because of the honesty, and support we give them.

Answer 2

Learn from it! If you've minimized the damage and found a solution, all you can do is try to use what you've been through to prevent it from happening again.

Also, you're better off freely admitting your mistakes, or even using them to educate others, than you are to hide your mistakes. Try to make something good come from it by letting others learn from you and prevent the same thing from happening to them.

Answer 3

I do not discover. Tester an managers discover it and share the joy of finding with me.

Answer 4

Just tell them up front that you made a mistake. That will actually increase your credability in the long run, because then they know you speak the truth. Nobody likes someone who tries to put the blame on someone else.

On the other hand, in some competetive environments there are people to will jump at this opportunity to bring you down. Avoid working in such environments.

Answer 5

First, as far as your customer is concerned, it shouldn't be your fault as an individual. Your team screwed up. It's none of their business which individual developer caused the bug.

Second, determine why that bug slipped past you. Didn't you test your code enough? Did you rush this feature because of deadline pressure? Are you working with error-prone code that could be made more robust? Should others have caught the bug in testing?

Everyone writes bugs. That's nothing to feel bad about. The problem is when the bug goes undetected. So who should have detected the bug? Even if you wrote it, it may not necessarily be your fault that it didn't get fixed. (Which doesn't mean you shouldn't take responsibility, of course. Just that it's not just you, but perhaps the entire team who have to stop and reconsider the process.)

Look ahead. Determine what can be done to avoid it happening again. What should you do while writing code, and what should your company do during testing, or when preparing for a release? What can be done in the code itself to make easier to work with, and harder to screw up?

Answer 6

Look at it as a eye opener, "shit happens", but why?

If you have accepted that you failed in some step you can either blame your self for being a bad person (not very constructive), or you could go back and look back and wonder how can I make sure this does not happen again.

Do I need to change my unit test, do I have unit test? etc etc

search this site for best practices and you will find a lot of thing that can help improve both yourself and your process.

But be aware. One time is nothing, but a couple off times is a lot harder to explain...

Cheer up Johan

Answer 7

Remember what Nixon learned:

It's not the crime, its the cover-up that gets you.

Be proactive about issues, yes. But also do not let the word "blame" get used by YOURSELF, the customer, or your manager. Once people start the "blame game" witch hunt things get much, much worse.

Don't let the blame game be played. You can do this by:

1. Accept responsibility (responsibility means you will make sure that the issue is resolved.)
2. Keep in front of the situation. You want to always be the first to know and tell your manager. If someone else tattles on you to the manager or customer, you want the information to be "old news".
3. Don't overpromise! Promise to keep the customer informed but don't commit to a fix date immediately.
4. If possible, be prepared to offer at least one (tested!) solution.
5. Give yourself time to verify a fix before telling customer about the fix.
6. Breathe deeply and slow down so that the fix is a good one, not another disaster.
7. Figure out how widespread is the damage. Only tell people proactively that really need to know ( avoid the public sackcloth and ashes )
8. Apologize once and then squelch repeated mea culpas. You want people to focus on how well you recovered - not that a mistake was made.

"Mistakes happened we have learned from the mistake. We need to focus on procedures to make sure these sort of mistakes get caught earlier. This is what I propose..."

Answer 8

Sounds like you already covered the bases with one exception.

Learn from this mistake, and do you best to ensure it doesn't happen again.

Like you said, we are all human, and mistakes happen. I ran $10 million through our order entry system one night, because I never changed the app.config in my unit tests, when we brought a new SQL server into production after a few weeks of testing in the development environment. ( $10 million out of a $30 million a year company.... people noticed ).

I sent out an email telling everyone to stop operations. Then I sent out a 2nd email stating what happened, why it happened, how I was going to fix it, and why it won't happen again.

That is the best I would ever expect from anyone