Which solution to represent large results ?

Question

Hi,

I'm working on source code analysis of the linux Kernel considering all the 2.6 versions...

I'm collecting results for each file compiled and for each version of the kernel...

So far, i'm using an ad-hoc format and then I analyze the data with a small python script (a typical result file is about 300MB)...

However, an ad-hoc is not really convenient, because it's hard to interpret the data, and not really flexible, since as soon as you change the format, your interpretation script must be modified too...

I would like something that can easily evolve and from which i can easily extract some information...

Which solution is the best in my case to represent the results ?

• ad hoc format ?
• XML ?
• DataBase ? (if so which engine ?)
• Google AppEngine ?

PS: i'm using Java 1.5.

Answer 1

I think the first comment from Oscar isn't too bad. Maybe you should rethink your format and the way of analyzing it. I wrote several ad hoc formats for analyzing data, too, just to realize that a different data structure (in my case a simple binary tree) could do the same task in seconds instead of minutes. Most of the time, a database might be the best solution and for a plain old Java Program you might consider hsqldb as the most convenient solution.