I have several large csv files with thousands of columns that I need to import and then remove entire columns based contents of the column. Is there an easy way to handle this in Ruby?
I could transpose the data and then just delete rows but I was wondering if there was a more syntactically sugary way of doing it.
You need to iterate on rows and remove columns with Array#slice!.
Something like:
my_array.each do |row|
row.slice!(3) if <insert condition>
end
should do it.
As you need to assign all the columns in the csv to the columns in the database you just need to ignore the columns you don't need in the csv. This blog posting has a good example.
A shot in the dark, but if the criteria is simple enough, and you can somehow sample the data to see what columns you need to remove, maybe you can skip importing the columns in the first place, if at all posible this should be quicker than slicing, which is the way to go otherwise.
I ended up using transpose and then using reverse_each to work up from the bottom row slice! on the rows that needed deleting. Thanks for the help.