How can I collect user feedback without physically seeing my users?

Question

Like probably a lot of software developers, I almost never see real users using my software.

It is, of course, quite difficult to get good user feedback in this situation. Even if some users agree to give me some piece of information about the way they use the software, there's a huge difference between how they really use it and how they think they use it.

By chance, my software is client/server, which means I can quite easily technically collect some information on the server.

Of course, nothing equals looking at a real user using the software in real life, but I think it's better than nothing, or at least it's worth trying :)

While I log all the exceptions raised on the client in my database, I've not been beside this point yet.

• Has anyone does that before?
• What information would you log?
• Are there some legal issues? How should I deal with those?

Answer 1

I face the same problem with the software I'm developing, though I have no users for it yet.

I generally think that monitoring should always be opt-in, and that you should have the ability to review before materials are being sent. I think most people would agree to that.

However, from a legal standpoint there are greater issues at stake. Some companies restrict users in installing software that has any components that "call home" for security reasons. Depending on the usage context, any monitoring data can potentially reveal secrets.

For example, my software annotates things in the IDE. If I transmitted "home" details about files that are open (rather than hashes), even without the content of these files, I would still possibly be sending confidential details. If your tool can be used to open images or documents, there may be similar issues.

I would suggest hashing or finding way of obfuscating results on the client side, and ensuring via sufficient tests that there cannot be a situation where your software sends information home without consent and obfuscation. If I'm not mistaken, if your software does so, even by mistake, you may be violating US federal laws.

Also, make sure to encrypt the details as you send them over the wire.

Finally, if some of your users are in the EU, where privacy laws are stronger, your database of exceptions may be legally considered a "database" in itself (e.g., if you store SQL statements as they were executed and failed and these contain production values). So you may have to follow a lot of the rules about personally identifiable information.

Answer 2

When I did UI development, I used to collect every user command (button push, menu selection) and log them to file with my own internal debug information, but auto-delete the log files after a few days. This information is invaluable when trying to debug your own software (user can rarely recall precisely the steps they took when a problem occurs). I also kept a record of every application startup, in case we had a compatibility problem with third party software.

The point is that the information wasn't used unless a problem did occur, it was kept locally with no remote access, and it automatically got deleted if there was no problem. Only if the customer called us in for a problem did we access the log data.

Actively tracking user operations and sending them back to base is a separate issue entirely, and I've always shied away from that.

Answer 3

This isn't exactly what you asked for, but you do have a few options that are not programming-related solutions:

1) Do some hallway usability testing (scroll down to #12).

2) Try a product like Morae to set up a more formal, but remote, viewing session.

3) Ask a client to watch over their shoulder, using something like GoToMeeting, CoPilot or WinVNC. Or go to their site for a day and hang out watching over their actual shoulder.

Any of these will give you a really good idea of what works and what doesn't.

Answer 4

You could do something like this, which captures mouse movements and replays them for you to see using javascript and ajax.