Reading and Parsing Files in .NET - Performance For Hire

Question

Hi,

I want the most performat way to read and parse a file.

Is it possible to read a file in .NET, but not load the entire file into memory? i.e. just load the file line by line as I parse the content of each row?

Does XmlTextReader load the entire file into memory or does it stream the file into memory as it reads the file?

Answer 1

I'm not sure about XMLTextReader, but you can read a file line by line using the FileReader objects. )

Answer 2

XmlTextReader works on a stream - so it does not read the whole file in memory.

Answer 3

You could use the ReadLine method of StreamReader Class:

string line;

// Read the file and display it line by line.
System.IO.StreamReader file = 
   new System.IO.StreamReader("c:\\test.txt");

while((line = file.ReadLine()) != null)
{
   Console.WriteLine (line);
}

file.Close();

For XML files I would go with XMLTextReader. See this article on Dr. Dobb's Journal:

"Parsing XML Files in .NET Using C#: Five different parsing techniques are available in .NET, and each has its own advantages"

Answer 4

What you can try is using the StreamReader.ReadLine function and test the performance compared to things like FileStream/TextReader.

Answer 5

I'm not sure about the XMLTextReader either, but you can read line by line like this:

Dim theLine as String
Dim fsFile As New FileStream(inputFile, FileMode.Open)   'File Stream for the Input File
Dim fsImport As New FileStream(outputFile, FileMode.OpenOrCreate)  'File Stream for the Output File
Dim srFile As New StreamReader(fsFile)  'Stream Reader for Input File
Dim swfile As New StreamWriter(fsImport)   'Stream Writer for Output File

Do While srFile.Peek <> -1              'Do While it's not the last line of the file
    theLine = srFile.ReadLine           'Read the line
    Messagebox.Show(theLine, "Line-by-Line!")
Loop

Answer 6

Dude, you should add the "Beginner" tag to your question too (I'm not dis'ing you, this is a beginner question). XmlTextReader does not load the entire file into memory, it operates on a stream.

All of filestream objects read the file in "chunks". You can specify how much data (i.e. how big a chunk) to bring back to your program with each call (i.e. m bytes, where m is any integer - possibly long - value, or with a text reader, a single line of arbitrary length) The OS will cache n bytes (where n is 0 some or all) per read for performance reasons. You have absolutely no control over the size of n, and will only frustrate yourself experimenting to find out what it is, as it changes due to a thousand different environmental factors.

Answer 7

here is a TextFileReader Class I have been using for years

http://www.dotnet2themax.com/ShowContent.aspx?ID=4ee44d6c-79a9-466d-ab47-56bba526534f