Best Programming Language/Library to analyze large data files ?

Question

Hi,

I'm collecting a quite big amount of data from program analyses (around 400 Mo / file and many files).

The files contain 4-5 kinds of records, each of these records contains 5-6 fields separated by spaces.

Now I'm using python scripts and generate charts using matplotlib. I'm not really comfortable with python (I don't find it really fast either) and I don't find the matplot library really flexible.

Do you know any good alternative or python is the best one ?

I want something fast and a flexible graphical library (output formats, kinds of charts).

thx.

Answer 1

Gnuplot? ROOT? R? Octave? Cernlib? SQLite?

Perhaps you can say more about the domain. Do just need to draw some graphs or do you need to process the data in some way first? Numeric processing or inter-connectedness or clustering or ...

Different tools have different strengths.

---

For just plotting one column of your data against another, gnuplot will work fine. Root, octave and R will all do more:

• Root is a nuclear/particle physics tool. Good for number crunching and complex decision making. Large, complicated, and very powerful. C++ based. My choice for most tasks, but I have it installed and know how to use it.
• Octave is intended as a matlab equivalent. Good reputation for crunching numbers.
• R is a statistics tool. Again, a good reputation.
• A RDBM would give you a different type of manipulative tools: a chance to look for different patterns in the data. SQLite is one of the easiest ones to set up, but there are others. You'd need a visualization layer.
• As for cernlib, that was a joke. It is what particle physicists used in the '80s and '90s (actually parts of it go back to the '60s). Powerful, but it shows its age. Fortran 77 based. I used it for my dissertation research.

Answer 2

For raw speed, you're going to have trouble beating something like C++, but you'll have to go another route if you want something with a nice easy to use graphical library. I'd use C++ to process the data into something more manageable (maybe a database), and then do the graphing with something simple to use. (maybe jasperReports)

PERL might do the trick if it's plain text, but again, if speed is enough to scare you off from Python, you probably won't be happy with PERL either.

Answer 3

I don't know what kind of charts you plan to produce, but have you thought about using a RDB like MySQL and a reporting package like BIRT?