Best practice for reading csv (with variable number of lines) into data structures

Question

Hey guys, I'm writing a small program to read in a csv with a variable number of lines and have a question about best practices:

Is the best way to create storage for the data on each line to make an array that holds the data structures of the csv (one per each line of csv)?

Size allocated to the array could be set to a large number (for example, more lines than there would ever reasonably be in the csv)? I have seen this in many examples on the web.

Or... is there was a smart way to tell how much space would be needed such as counting the lines before hand or dynamically adding space by using a linked list as opposed to an array with static storage allocation. Any best practices? I don't think choosing a random number seems very slick...

Any thoughts would be greatly appreciated. Thanks!

Answer 1

Two best practices:

1. Never expect input from the outside to correct.
2. Make it transactional (imports all or rolls back)
3. If possible, leverage a third-party API or library like this http://www.codeproject.com/KB/database/CsvReader.aspx or this http://sourceforge.net/projects/javacsv/ to greatly reduce reinventing to wheel. If you're sticking to C and can do C++ consider this approach: http://stackoverflow.com/questions/415515/how-can-i-read-and-manipulate-csv-file-data-in-c

Answer 2

Use a library or count the lines beforehand. You could also use some kind of list data structure to avoid worrying about the line count.

+1 to Nissan Fan for recommending a library, in my opinion, unless you're trying to learn a lot about CSV parsing and CSV parsing edge cases, this is always the way to go.

Answer 3

There's really no "best practice". Keep in mind the particular structure of your data, how quickly you want to read it, store it, query it, sort it, find/eliminate/ignore duplicates, etc. A tree, a linked-list, hashing, ordered data, etc. are good options depending on the factors that I already mentioned.

I agree with the other fellows. No need to reinvent the wheel. There must be gazillions of samples about how to parse CSV.

However, when choosing your favorite library, a few words of caution:

1. Best practice: Never assume that the data has a specific (small or very large) amount of data. Corollary: don't store all the data in memory, just as little as reasonable, and assume that whatever the size of your array, the data may be bigger than it. With that being considered, work around that assumption.
2. Another best practice: Test corner cases (no input, very large input, only one line or element, etc.)
3. CSV files are not standard. For example, some programs that generate CSV just ignore the following cases:

3.1. Commas within strings. For example, it's not the same "Smith, John" than Smith, John. 3.2. Special characters withing the strings, such as apostrophes, tabs, or quotes. How are they handled? For example, Microsoft typically uses double-double quotes to represent quotes inside a string. 3.3. And, of course, be careful with the end of line format (Unix or Windows-style).

Be sure to take a look to very good bunch of actual data. Never believe the users (nor the programmers :-).

Good luck. Luis. Excel and Visual Basic used to generate

Answer 4

If you can process the data as you read it rather than saving it all and processing after, this would eliminate the problem.

I avoid counting the lines first, as this requires reading the entire file twice. I suppose if the file is small the efficiency hit is not a big deal, but if you know that the file is small, then you could just allocate a big enough space.

So in general, my approach -- if I can't process the file a line at a time -- is to use a data structure that can grow, like a linked list. Then for each line I just allocate a new block. Depending on what you're up to, you might use a dynamic array: allocate an amount of space that ought to be enough for the normal case. If you fill it, allocate a bigger space, copy the first to the second, delete the first, and then continue working with the second. If you fill that, repeat the process. This can be a lot of data movement but the amount of space used in the end will be less than a linked list because you don't have the pointers, and it will be faster to traverse because you're not chasing pointers and possibly running all over virtual memory.