Adding new "columns" to csv data file in Tcl

Question

Hi All,

I am dealing with a "large" measurement data, approximately 30K key-value pairs. The measurements have number of iterations. After each iteration a datafile (non-csv) with 30K kay-value pairs is created. I want to somehow creata a csv file of form:

Key1,value of iteration1,value of iteration2,...
Key2,value of iteration1,value of iteration2,...
Key2,value of iteration1,value of iteration2,...
...

Now, I was wondering about efficient way of adding each iteration mesurement data as a columns to csv file in Tcl. So, far it seems that in either case I will need to load whole csv file into some variable(array/list) and work on each element by adding new measurement data. This seems somewhat inefficient. Is there another way, perhaps?

Answer 1

Because CSV files are fundamentally text files, you have to load all the data in and write it out again. There's no other way to expand the number of columns since the data is fundamentally row-major. The easiest way to do what you say you want (after all, 30k-pairs isn't that much) is to use the csv package to do the parsing work. This code might do what you're looking for…

package require csv
package require struct::matrix

# Load the file into a matrix
struct::matrix data
set f [open mydata.csv]
csv::read2matrix $f data , auto
close $f

# Add your data
set newResults {}
foreach key [data get column 0] {
    lappend newResults [computeFrom $key]; # This is your bit!
}
data add column $newResults

# Write back out again
set f [open mydata.csv w]
csv::writematrix data $f
close $f

You would probably be better off using a database though. Both metakit and sqlite3 work very well with Tcl, and handle this sort of task well.