Is there a "proper" way to read CSV files

views:

1101

answers:

+9 Q:

Is there a "proper" way to read CSV files

In .net, is there a standard library that should be used to read in csv files? All the samples on the web roll their own csv reader / parser, or use OleDb.

It's not a problem using any of these solutions, I was just wondering if there is a generally accepted library (not that I can find), or any other "proper" way to do it?

I'm pretty sure you can read a CSV file into a DataTable with one line of code. Once it's in a DataTable, you can sort, filter, iterate etc.

This question has some examples for reading CSVs into DataTables.

MusiGenesis 2009-07-09 12:28:22

I have been down this path many many times. When it works, it works fine - but debugging problems when it doesn't work is a massive pain.

aSkywalker 2009-07-09 13:12:28

Huh? The only problems I've ever run into with CSV files were with the data itself (extra commas, missing quotation marks etc.), and those would be problems no matter what parsing method you use.

MusiGenesis 2009-07-09 13:34:48

yes, the problem is almost always with the data - but where in the data? We started switching over to the TextFieldParser class in every spot (hundreds) because of the limitations and lack of control it gave. Worked 90% of the time, but when it didn't there was no help provided in the error. We created a class library to work around this - scrubbing the raw data first etc... If you haven't tried the TextFieldParser class you really should - we love it - and parse millions of rows of csv data each month

aSkywalker 2009-07-10 12:05:02

+12 A:

CsvReader is a pretty good one... it isn't Microsoft, but it works very well, and is a lot faster than some of the alternatives (legacy OleDb etc).

Marc Gravell 2009-07-09 12:31:56

This was the best library I could find. I had to fix a few bugs in it related to more obscure support for certain things but it was fine on the standard stuff. IT is somewhat inefficient in that it does repeated string concatenation. With a bit of rejigging I changed it to use StringBuilder in the main loop for most cases and that gives you a massive speed boost

ShuggyCoUk 2009-07-09 12:48:29

Yes, I'm investigating it at the moment - it seems to handle a lot of the edge cases. There doesn't seem to be anything else like it.

Gareth 2009-07-09 12:50:07

I'm actually going to see if the guys wants my code changes rather than trying to maintain it myself...

ShuggyCoUk 2009-07-09 12:58:56

+5 A:

lavinio 2009-07-09 12:36:41

These all sound like good reasons *not* to write your own, unless you want to repeat the same mistakes that others have already made (and possibly fixed).

LukeH 2009-07-09 12:57:45

John Machin 2009-07-10 03:31:47

I agree. This is why I was encouraging people *not* to write their own, or if they *have* to, to think about the issues.

lavinio 2009-07-10 03:35:46

@lavinio: You gave what you regarded as reasons why people write their own. There is nothing in your answer (even after your edit) that could in any way be construed as encouragement to do otherwise.

John Machin 2009-07-10 04:57:32

I don't think you can properly make the decision until you understand what processes are going to be sharing the data. Ideally, you'd use a prepackaged one, but I've spent enough time trying to solve incompatibilities with this "simple" format to know otherwise. (I work for a data integration company.)

lavinio 2009-07-10 14:06:35

+2 A:

After some more investigation, there is also this: http://www.filehelpers.com/

It seems a full framework around reading files, and not just csv files.

(note: just read stuff on the website, have not used it yet)

Gareth 2009-07-09 12:58:34

+3 A:

The VB namespace has a great TextFieldParser class. I know, c# people don't like to reference a library from that 'basic' language, but it is quite good.

It is located at Microsoft.VisualBasic.FileIO.TextFieldParser

I used to mess with OLEDB, creating column definition files etc - but find the TextFieldParser a very simple and handy tool for parsing any delimited files.

aSkywalker 2009-07-09 13:11:35

Yes, I like this class too... but I really wonder why MS put it in a VB-specific assembly, it doesn't make any sense !

Thomas Levesque 2009-07-09 13:24:12

@Thomas: VB programmers expect easy-to-use string parsing functions, whereas C-style programmers expect to suffer horribly when it comes to strings.

MusiGenesis 2009-07-09 13:31:43

KBCsv is another option, particularly if you require efficiency and the ability to work with massive CSV files.

HTH, Kent

Disclosure: I wrote KBCsv, hence the "KB" ;)

Kent Boogaart 2009-08-04 12:28:54

ansaurus

tags:

views:

answers:

Is there a "proper" way to read CSV files

related questions