Troubleshooting user issues - techniques?

Question

This is a pretty general question - but what are some of your strategies / procedures for troubleshooting issues that users find?

Specifically, I'm talking about web based programming - a combination of PHP, mySQL, Javascript (Ajax).

We're running an ecommerce platform and we've done successful test transactions on all major browsers. We've done in person user testing to discover bugs and study user patterns. And we've processed hundred of orders without a hitch. But of course, every once in a while a user will come across a bug we've never seen before.

The frustrating thing is that often we can not recreate the bug, so its hard to try and fix it. It's also frustrating going back and forth with a customer trying to work out our technical glitch or determine if they're doing something odd...while potentially frustrating them and loosing an order. Although, fortunately, I've found if we treat them super kindly, we get nothing but kind and appreciative replies.

So what do you do when a user writes in about a bug you can't recreate? Do you have any tips on how you isolate the problem, the steps you take? etc.

Answer 1

• Don't believe what the user thinks they did. They will forget or never even notice lots of critical information.
• Be careful with what meaning you apply to the terms they use (they don't necessarily know how you use terms)
• If possible, watch them repo the issue.

If I found a bug in a web site and their dev's called/emailed me asking for feedback ("could we watch you repo the bug with <some sort of shared screen system>?") I would be highly impressed.

Answer 2

• Ask the user for Fiddler logs (or any other HTTP sniffer)
• Don't ask the user what they did. Find a recording tool that allows you to record the user actions and ask the user to run it while reproing the bug. Ideally, it would be a tool that records mouse clicks and keyboard presses and takes screenshots at that point of time. Unfortunately, we use an internal tool for this, so I can't point you to a download link.
• Get as much details about the user's OS, browser and installed browser plugins (the last part is especially important for IE, as there are quite a lot of BHOs out there that mess up with the page). This would potentially allow you to narrow the scenario and make it easy for you to repro the problem locally.

Answer 3

Get as much information as you can (what the user was doing, what exactly happened) and then don't trust 90% of it (since users can be either less than forthcoming, or clueless. Browser (including version) can be key with some issues, particularly AJAX.

Go and talk to the user (if possible). Or try to do a GoToMeeting or similar screen sharing system to watch what the user is doing. Also, if you can access the browser, you can check for stuff like caching, script settings, etc. This is particularly useful in light of the fact that user testimony is sometimes flimsy (the application "blew up" doesn't tell me much).

Never jump to obvious conclusions (as they can often-times be wrong). Be thorough, and try to follow the same set of steps each time. Returning users will learn our drill, and you'll have a consistent way to track down problems.

Answer 4

With a custom solution/framework that I used once upon a time the framework logged every transaction between the server and the client so we had a record of every transaction that occured, including the values of form. That meant we essentially new exactly what the user actually did.

That proved to be invaluable when it came to figuring out what happened. We could look through the logs and look at the exact values that were entered. As has been mention this never really matched quite what the user said they did.

Unfortunately this framework had to be written and that is why it is not possible in systems like SharePoint for example.

Answer 5

If I could +3 BCS for those suggestions I would.

Remote assistance tools like InQuerio, WebEx, etc work wonders for actually seeing what a user is doing and are minimally invasive.

---related but preachy, read on at your own peril / enlightnement---

Some advice that 10 years on help desks has given me. This is the key tenet to resolving customers issues that is so often overlooked.

Tech Support == Customer Service

People skills is as much a part of support as is technical acumen.

Treat them as professionally and as patiently as possible and dont lose your cool if the problem or the customer is being tough. Be properly confident and don't appear ineffectual, they want someone to help them fix the problem.

Also, try to make it as easy as possible for them. Remember that to the user, this is your problem and they shouldn't be inconvenienced (any more than is absolutely necessary) with installing or configuring any extra software (fiddler -gee teach them firebug too while yer at it, PcAnywhere, VNC, etc.) or performing seemingly outlandish tasks.

Answer 6

Get as much info as possible about user's environment. For IE use IE - Help - About - System Info.

See what's on the wire - use e.g. WireShark

Hopefully you have some kind of logging package where you can configure debug levels and amount of logging. Set to "Debug / Verbose"

If the user is using FireFox, ask them to install TestGen4Web and record the session for you to playback later.

Answer 7

Good suggestions.

In addition, if all your efforts to repeat a problem fail, at least keep a record of it, and trend it. One occurrence of a freak issue can sometimes be dismissed, but if it pops up more than once, it's worth investing serious effort investigating it.

Answer 8

If the customer allows it, we'll often use a desktop-sharing utility or service such as webex to watch the customer use the product. That allows us to see the details that aren't always communicated over the phone.

Answer 9

If you analyze the "difficult" bugs you've encountered so far, you may be able to find some commonalities between them. This information can be used both to help you improve your troubleshooting process and to teach your developers to make their code more robust, or at least leave more "breadcrumbs" that you can follow.