How to elicit debugging information from end-users?

Question

During beta testing, sometimes our end-users unfortunately encounter a bug. This inevitably leads to a bug report, where the Joel on Software article Painless Bug Tracking suggests:

Every good bug report needs exactly three things.

1. Steps to reproduce,
2. What you expected to see, and
3. What you saw instead.

This is usually pretty easy to enforce internally, however end-users often only send an email of what they saw (it crashed! Or it generated the wrong result!). They don't always report what they expected to see and more importantly the steps to reproduce the problem. Log files can help, but typically do not provide a complete picture of what caused the error. After a recent discussion with an end-user to try and discover what led to a problem, it got me thinking. How could I do this better? Should I have some sort of conversational structure? What are the best questions for gathering information from a layperson? Or do I need a completely different approach to elicit debugging information from end-users?

Answer 1

When an unexpected error happens, pop up a dialog with the text "Sorry, an unexpected error happend. Please click the button below and paste the log into your error report" plus a button "Copy log to clipboard".

Now you can include all the information you like in the error report. This has the additional advantage that the user can see what the error report contains, so they can remove any confidential information.

Answer 2

Assuming you've communicated the three required things with your end users, change your approach. Here's an interesting question for you:

Why much is a bug worth to you and your team?

Is it worth writing a blog post dissecting a good bug and thanking the people who submitted it? Is it worth thanking the customer publicly in your release notes? Is it worth sending that customer a thank you gift of cookies, fruit, etc?

Make the value of submitting good bugs tangible to everyone involved and behaviors will change. The key though is to recognize individuals. Stroke their egos just a bit.

If you're already set expectations, it's up to you to celebrate and reward people who have done it successfully. And the best thing is that you only have to do this a few times (once/month?) before things start improving. If you keep it up for a whole year, you might have something interesting to talk about on other fronts.

Answer 3

Having a set of structured questions will help but all too often the users won't really have been paying attention.

"So I just got an error message.

"What did it say?"

"I don't know, something about an error in a module. I just clicked OK."

"OK, what were you doing?"

"Just updating some records."

"Which record was it?"

"I don't know, I've done a bunch this morning."

All too often the good users will give you the information you want, the bad users won't regardless of how you ask.

Personally I'd go down the better logging route. You can't log every action all the time but you can have detailed logging which can be enabled for a particular user. This would give you a detailed stack trace with IDs and parameters, as well as logging any errors, and potentially gather information about the system, versions and so on (in the bad old days we had one that did a complete DLL version sweep).

This also helps with things which aren't obvious "errors", that is where things just seem to be wrong but there are no dialogues popping up.