Parsing a CSV File to a Rails Database

Question

G'day guys,

I'm using fasterCSV and a rake script to parse a csv with about 30 columns into my rails db for a 'Trade' item. The script works fine when all of the values are set to strings, but when I change it to a decimal, int or other value, everything goes to hell.

Wondering if fasterCSV has built in int etc parsing or whether I'll have to manage these within my model.

Basically, I'm given a giant amount of trades data, need to import it, and then need to provide feedback with say the average trade volume, the times, etc. I understand I can do that all with the wonderful records provided to me by activeRecord but wondered if there was an easier way to populate a rather large Database with a given CSV?

Several of the fields don't have values for certain rows, fasterCSV seems to work perfectly when they're all strings, but not when I try to get decimal or other.

Answer 1

Assuming you're using MySQL, you should be able to import the CSV directly into the database with LOAD DATA INFILE.

Writing a Ruby script to do this is kind of reinventing the wheel, as there are many, many tools out there that do this already.

Answer 2

Have you tried to use AR Extensions for bulk import? You get impressive performance improvements when you are inserting 1000's of rows to DB. Visit their website for more details.

Refer to these examples for more information

Usage Example 1

Usage Example 2

Usage Example 3