Library for an interactive stats website

Question

How would you go about coding an interactive website to display stats/graphs. Say I wanted to create something interactive for people to look at Stackoverflow stats - something that looks like awstats / google analytics but allows you to drill down to stats/graphs like:

• All questions: total, by hour of day, by day of week (interesting timezone challenge there, or just stick to UTC).
• Tags (e.g C# questions, app-engine questions): totals, by hour of day, by day of week
• Select a user: totals, by hour of day, by day of week
• Extra cool: the ability to add x number of users / tag, date ranges.

Is the answer "code it yourself"? I guess I could pre-crunch a lot of data and find a library to create the graphs for me.

Or is there a library/package suited to this sort of thing? I've spent some time looking at datamining applications (Splunk, SQL Server Analysis Services). But these look like interactive applications to build up queries, not something to create interactive output.

I'm not attached to any particular web tech. ASP.NET / PHP / python are all good.

Answer 1

I would suggest starting by taking a look at the code behind the open source Google Analytics alternative Piwik.

Answer 2

There are projects that attempt to integrate graph-drawing with data collection, storage and analysis (such as Cacti), but if I were doing this I'd roll my own data management code and use an external language/library/program to make pretty graphics.

If you're interested in doing that, I'd look at the following:

• matplotlib - a library for Python that allows you to make pretty graphs.
• R - a language that has functions for data analysis and drawing pretty.
• gnuplot - an open source program for plotting data.

All of these can generate graphs for you given the data, but none of them take care of storing the data for you. If you want to see some beautiful graphics made with R, check out this web site:

R Graph Gallery

Answer 3

1. Grab dump from here: http://blog.stackoverflow.com/2009/06/stack-overflow-creative-commons-data-dump/

2. Process into your favorite DB.

3. Make pretty using Google Chart API. Cool DB integration example here.

Answer 4

Maybe take look at http://www.kapott.org/~kleiner/interactiveStats. There is currently only a small demo page available. AFAIK there is no release until now, and the demo page still lacks some features like graphical output etc. But work is in progress, and I'm looking forward for the first release... :-)