What's the best way to read a tab-delimited text file in C#

Question

We have a text file with about 100,000 rows, about 50 columns per row, most of the data is pretty small (5 to 10 characters or numbers).

This is a pretty simple task, but just wondering what the best way would be to import this data into a C# data structure (for example a DataTable)?

Answer 1

I would read it in as a CSV with the tab column delimiters:

A Fast CSV Reader

Edit:
Here's a barebones example of what you'd need:

DataTable dt = new DataTable();
using (CachedCsvReader csv = new CachedCsvReader(new StreamReader(CSV_FULLNAME), false, '\t')) {
    dt.Load(csv);
}

Where CSV_FULLNAME is the full path + filename of your tab delimited CSV.

Answer 2

Two options:

1. Use the classes in the System.Data.OleDb namespace. This has the advantage of reading directly into a datatable like you asked with very little code, but it can be tricky to get right because it's tab rather than comma delimited.
2. Use or write a csv parser. Make sure it's a state machine-based parser like the one @Jay Riggs linked to rather than a String.Split()-based parser. This should be faster than the OleDb method, but it will give you a List or array rather than a datatable.

Answer 3

What about FileHelpers, you can define the tab as a delimiter. HEad on over to that site by the link supplied and have a peeksy.

Hope this helps, Best regards, Tom.

Answer 4

However you parse the lines, make sure you use something that supports forwarding and rewinding, being the data source of your data grid. You don't want to load everything into memory first, do you? How about if the amount of data should be ten-fold the next time? Make something that uses file.seek deep down, don't read everything to memory first. That's my advice.

Answer 5

Simple, but not the necessarily a great way:

• Read the file using a text reader into a string

• Use String.Split to get the rows

• use String.Split with a tab character to get field values

Answer 6

Use .NET's built in text parser. It is free, has great error handling, and deals with a lot of odd ball cases.

http://msdn.microsoft.com/en-us/library/microsoft.visualbasic.fileio.textfieldparser(VS.80).aspx