Observing or monitoring users working with an application remotely

Question

I'm a believer in observing what users are doing with an application. I think that it is the only way to get an accurate picture of what people are doing. However, I don't always want to be sitting with them and peering over their shoulder; apart from the time burden it is a distracting for them and may also change their behaviour.

What I'd really like is a way of observing/recording/monitoring/logging what users are doing in a windows form application.

I'd also like to be able to analyse the actions of 50+ users and gather some stats.

And the observation is like this http://www.useit.com/alertbox/20010805.html not like spying on people.

Got any suggestions?

Answer 1

Do you want a log of the actions? Or do you just want to observe?

Perhaps simply DrawToBitmap on a timer? (although this is a bit hit'n'miss; WPF has better tools here...)

Sometimes this isn't evil... we (actually, Steve) did something similar on the finguistics prototype to mimic a teacher's console (for tracking activity etc).

Answer 2

You can take a look at oDesk's small tool. They provide such a tool for monitoring oversea freelancers. The tool will capture screen shot with random interval, count the keyboard event number and mouse event number for every minute. Thus you know if he is working.

Answer 3

You could use TimeSnapper, it takes a screenshot at configurable intervals and allows you to play back the screenshots like a movie. The free version does what you need. You can configure it to snapshot the whole desktop or just the active window.

For real-time observation you could use VNC which is again free.

Edit: for playing back the Timesnapper files from the remote PC, copy them from the configured snapshots directory to your own. You could also of course use Timesnapper to take screenshots of your VNC window, to capture the observation locally.

Answer 4

With this type of implementation, please provide some sort of privacy policy, notification, and dialog to ask the user's permission! As Marc Gravell stated, you can put DrawToBitmap on a timer and only capture your application window. These image may be large depending on your application, so that's a large consideration if you are transmitting over a network. I would look into some third party libraries possibly instead of reinventing the wheel here, as I did that with a remote desktop application, and while I got it to run at well over 60FPS using GDI+, I failed miserably at getting packet size down to a minimum so I could transmit over the internet.

Answer 5

If you want to 'see' what they're doing, then the suggestions about screen-grabbing at intervals is probably your best bet. This will allow you to see HOW they do something (based on what they expect the UI to behave like), but not necessarily WHAT they are trying to achieve.

However, I would say that a log is probably a better method to see WHAT people are doing. If your log is detailed enough, you can tie every action to an event trace, and build up a time-line and kind of stacktrace of what the application did as a result. From this (and presuming your software UI is designed fairly well), you can see exactly what they were trying to do, but the details of HOW they did it have to be inferred (again, this varies depending on how detailed your log is).

Answer 6

No need to develop your own solution to this problem. There are quite a few commercially available apps for this.

• Moderated remote usability testing: where you watch the user via VNC style streaming footage
• Unmoderated remote usability testing: where users do the testing on their own, quantitative data is gathered.

Check out http://remoteusability.com/ for more information

Answer 7

Disclaimer: I am a developer on the products that I am going to mention.

If you are looking for a tool that will let you know what a user is doing with your application but do not need actual screen captures you can look into using Runtime Intelligence from PreEmptive Solutions.

What we do is to inject additional code into your application after it is built (similar to PostSharp's IL Weaving) that sends data back to a central collection and reporting portal whenever methods that you decorate with attributes are executed. This allows you to track when users are using your application, which of the decorated methods are executed (and it can also give you measurements of how long the decorated methods take to execute) and if the user exited your application as expected or if there was an error. The data can be sent back either to servers hosted at PreEmptive (both a free and commercial version) or to any URL of your choosing if you want to capture and store the data yourself.

Since this relies on using Dotfuscator as the code injection engine this functionality can be added to any .NET application (console, Winforms, WPF, Silverlight, etc) as a post build step. We provide both a set of custom attributes if you want to decorate methods in your source or you can use our user interface to can specify which methods will be instrumented and that data will be stored inside of the Dotfuscator project file. If you use the Extended Attributes feature of storing the injection points in the project file you can completely instrument an application without touching the original source code.

We provide a hook so that you can give your user the choice of opting in or out of the usage tracking. Since we work at a low level this does require that you write a method, property or field that contains a boolean value indicating if data should be sent that we check at runtime. You are responsible for actually creating the UI for opt-in/opt-out.

A free version of all of this, including a freely accessible data reporting portal, is going to be available in Visual Studio 2010 as part of Dotfuscator Community Edition. You can go ahead and download Visual Studio 2010 Beta 1 and try it out today if you wish.

The free functionality is a subset of what is available in the commercial version but it will give you a good idea of how easy it is to use. As always, PreEmptive is happy to provide you with a free, time limited evaluation version of the commercial editions so you can test out the unlimited functionality version.

I am currently writing a series of blog articles on using this as part of Visual Studio 2010, the first one is here and an overview of everything coming in Visual Studio 2010 Beta 1 is here.

Runtime Intelligence is also available for use on any Java application by using DashO for Java as the code injection platform. There is currently no community version of this although there is always the time limited evaluation version.