What's the best interactive Analysis and Plotting Tool for software testing?

Question

My realtime app generates a data log: 100 words of data @10Khz. I need to analyze it and produce some plots of the results. There are intermediate calculations involved - I need to take some differences, averages, etc. Excel would work fine, except for:

• the 32000 item limit on graph data series is too small - that's only 3 seconds of data.
• the glacial speed at which it processes changes to graphs containing large data series is unbearable.

What are good alternatives to Excel for manipulating and plotting large quantities of data? I'm looking for something interactive, not a library.

Answer 1

For this sort of stuff we typically roll our own, but I know that isn't the solution you want. Can you use a good quality database (eg Oracle) to do the manipulation, then maybe put the summarized data back into Excel for the plotting? I believe Excel will link to databases these days, so you could make it quite automated.

Otherwise there are statistical tools like [SAS][1], but get your cheque book out first.

[1]: http://www.sas.com/technologies/analytics/statistics/stat/index.html SAS

Answer 2

There are also several free tools for analysing and plotting (see below). But I am not sure whether they have components to handle data in real-time.

1. R (similar to SAS) for statistical computations
2. octave (similar to Matlab) for mathematical computations

Answer 3

In particle and nuclear physics the big tool is ROOT, which I have seen used in a "update every two seconds as the data comes in" mode with a lot of data and a modest amount of intermediate processing.

Mind you, the student who wrote that module was a very slick programmer, and it took a while to shake the bugs out, even so.

ROOT is available for free, and provides all kinds of tools and support.

Answer 4

R (for data manipulation) and its ggplot2 module for creating sexy graphs. Incredibly useful.

If you need real-time graphics, then I'd look at building something using matplotlib. It's a Python module, and you can link it to R using rpy2 if required.