What is the best method to gather data about the use of your application?

Question

My company releases a small software product for which I've recently been taking over the development side. It is a C# Windows Forms application.

One of the things I've noticed is that much of the information about how the software is used is filtered through my superiors and I get the feeling that I'm missing important detail in some of the messages.

I realise I'll have to work on the management issues with this situation, however in order to give another view on the problem I've been considering a technological solution. Perhaps something similar to the "Microsoft Customer Experience Improvement Program".

I was wondering if anyone out there had any experience or advice monitoring and reporting on user behaviour in their applications?

Answer 1

I would suggest you get your application to write its "usage information" somewhere and then, with the users permission, transmit it electronically every so often.

Note the emphasis above. Depending on your jurisdiction, you could get into serious trouble transmitting any sort of data from someone else's computer without permission.

You're more likely to get permission if:

1/ You make it clear, on install or update, that your program will collect information and transmit it only with permission.

2/ You explain clearly what the information is and that it only holds "usage information", nothing that can be traced back to the user of the software (NO serial numbers, etc).

3/ You request permission to transmit infrequently. If I had an obnoxious program that asked me daily, I'd soon stop using it altogether.

Answer 2

Coincidentally, I'm currently working on a platform for collecting anonymous usage statistics from desktop applications.

Here's a screenshot:

http://benjismith.net/index.php/gui-mockup-version-1/

And here's a blog post where I talk about the idea:

http://benjismith.net/index.php/2008/06/02/business-intelligence-for-desktop-software/

I hope to have something on the market within the next two or three months :)

Answer 3

I believe that you've already received a good answer to your question re the privacy aspect of the technological solution.

I would also like to mention that you should try to use HTTP or HTTPS over normal 80/443 ports - these are the least likely to cause problems with the firewalls and proxy. Use MS IE proxy settings as they are usually set properly.

---

From a totally different prospective I would like to say that the best way to learn about the usage of your software is to check if you have any 'friendly' users in your install base and interview them. It could be some partner company or people who are your 'pilot group' for the betas of new release. Talk to them. Grap your manager and spend a day going to their site if possible and just seeing for yourself how and then they click the buttons. Make notes. Listen to the feedback. Establish personal relationship so next time they talk to you directly.

If your application is small and your company is no Microsoft its much better to be close to your userbase than to collect anonymous usage data which takes significant effort to collect and process and then even more effort to understand.

Answer 4

One idea is to send anonymous statistics with your users' permission.

Another idea is to provide a big report bug/ask question button in your app so they can tell you when something they think is wrong happens you'd send app state along the report.

Always be clear on what are you going to be sending and make users have an option to be non anonymous (always set anonymized data as default) and you might be surprised when you get a lot of non anonymous data by choice of users.

And be clear by using users' language, NEVER say things like "I'm going to send a Blowfish encrypted memory dump of the current state of the application's stack and heap. Yes/no?", but things like "I'm going to send a list of your activity in the program: the buttons you clicked and the type and amount of files you opened. This will help us to create a better program for you but you can choose what you want us to receive. "

Answer 5

Disclaimer: I am a developer on this product, so I may be a bit biased on how great it is :)

There is currently a product on the market that can provide you with this functionality for both .NET and Java applications that we call Runtime Intelligence. See: http://www.preemptive.com/runtime-intelligence-services.html for details.

This product is currently shipping for both .NET and Java and a free version offering a limited feature set will be included in Visual Studio 2010.

Usage of an application can be tracked ranging from high level statistics on number of times executed and on which Operating System/Framework versions down to the frequency of usage of individual features.

We provide a way for you to give your users either an opt-in or opt-out choice in the transmission of the usage data and the data is sent either over SSL (the default) or standard HTTP. The performance impact on your application is minimal and we take great care to not impact the responsiveness of your code.

This solution is great for evaluations or beta testing as you can track the usage of your application directly rather than relying on surveys or guessing at what the users are doing.

For released applications this provides the scaffolding of a customer experience improvement program and the visibility into the accumulated data is valuable to everyone from upper management down to the developer.

We have found that it takes less than an hour to set up an application for basic usage reporting with no code changes required since we can work directly on the assembly binaries.

For pricing information or to obtain an evaluation please contact sales from our website as I'm just a developer :)

For detailed technical information or any other questions feel free to contact me.

Answer 6

Actually, I think you're trying to do simple Business Intelligence.

Don't forget to set the "appropriate" dashbord in order to track your collected data, think deeply on the indicator you'll implement.