What is the optimal (speed) way of parsing a large (> 4GB) text file with many (milions) of lines?

Question

I'm trying to determine what is the fastest way to read in large text files with many rows, do some processing, and write them to a new file. In C#/.net, it appears StreamReader is a seemingly quick way of doing this but when I try to use for this file (reading line by line), it goes about 1/3 the speed of python's I/O (which worries me because I keep hearing that Python 2.6's IO was relatively slow).

If there isn't faster .Net solution for this, would it be possible to write a solution faster than StreamReader or does it already use complicated buffer/algorithm/optimizations that I would never hope to beat?

Answer 1

StreamReader is pretty good - how were you reading it in Python? It's possible that if you specify a simpler encoding (e.g. ASCII) then that may speed things up. How much CPU is the process taking?

You can increase the buffer size by using the appropriate StreamReader constructor, but I have no idea how much difference that's likely to make.

Answer 2

Do you have a code sample of what your doing, or the format of the file you are reading?

Another good question would be how much of the stream are you keeping in memory at a time?

Answer 3

A general note:

1. High performance streaming isn't complicated. You usually have to modify the logic that uses the streamed data; that's complicated.

Actually, that's it.

MSN

Answer 4

Sorry if I'm not a .NET guru, but in C/C++, if you have nice big buffers, you should be able to parse it with an LL1 parser not much slower than you can scan the bytes. I can give more detail if you want.

Answer 5

Try BufferedReader and BufferedWriter to speed up processing.

Answer 6

The default buffer sizes used by StreamReader/FileStream may not be optimal for the record lengths in your data, so you can try tweaking them. You can override the default buffer lengths in the constructors to both FileStream and the StreamReader which wraps it. You should probably make them the same size.

Answer 7

If your own code is examining one character at a time, you want to use a sentinel to mark the end of a buffer or the end of file, so that you have just one test in your inner loop. In your case that one test will be for end of line, so you'll want to temporarily stick a newline at the end of each buffer, for example.

The Wikipedia article on sentinels is not helpful at all; it doesn't describe this case. You can find a description in any of Robert Sedgewick's algorithms textbooks.

You might also want to look at re2c, which can generate very fast code for scanning text data. It generates C code but you may be able to adapt it, and you can certainly learn the techniques by reading their paper about re2c.