Reading large csv files

Question

Which is the most performant way to read a large csv file in .NET? Using FileStream? or another class? Thanks!

Answer 1

You can use a stream but read the file line by line

Answer 2

If you want to read it all into memory, a simple File.ReadAllText() will do just fine.

EDIT: If your file is indeed very large, then you can use the StreamReader class, see here for details. This approach is sometimes inevitable but should mostly be avoided for style reasons. See here for a more in-depth discussion.

Answer 3

You can use the StreamReader returned by FileInfo.OpenText:

Dim file As New FileInfo("path\to\file")

Using reader As StreamReader = file.OpenText()
    While Not reader.EndOfStream
        Dim nextLine As String = reader.ReadLine()
        ProcessCsvLine(nextLine)
    End While
End Using

Answer 4

The most efficient way of doing this is by taking advantage of deffered execution in LINQ. You can create a simple Linq-To-Text function that read one line at a time, work on it and then continue. This is really helpful since the file is really large.

I would desist from using the ReadBlock or ReadBlock or ReadToEnd methods of StreamReader class since they tend to read a number of lines at once or even the entire lines in the file. This ends up consuming more memory than if a line was read one at a time.

public static IEnumerable<string> Lines(this StreamReader source)
{
    String line;

    if (source == null)
        throw new ArgumentNullException("source");

    while ((line = source.ReadLine()) != null)
    {
        yield return line;
    }
}

Note that the function is an extension method of the StreamReader class. This means it can be used as follows:

class Program
{
   static void Main(string[] args)
   {
       using(StreamReader streamReader = new StreamReader("TextFile.txt"))
       {
            var tokens = from line in streamReader.Lines()
            let items = line.Split(',')               
            select String.Format("{0}{1}{2}",
                items[1].PadRight(16),
                items[2].PadRight(16),
                items[3].PadRight(16));

       }
   }
}

Answer 5

I had very good experience with this library:

http://www.codeproject.com/KB/database/CsvReader.aspx