Max file size for File.ReadAllLines

Question

Hi, I need to read and process a text file. My processing would be easier if I could use the File.ReadAllLines method but I'm not sure what is the maximum size of the file that could be read with this method without reading by chunks. I understand that the file size depends on the computer memory. But are still there any recommendations for an average machine? I would greatly appreciate your fast response. Thanks, Lev

Answer 1

I've done stuff like this with 1-2GB before, albeit in Python. I do not think .NET would have a problem, though. But I would only do this for one-off processing.

If you are doing this on a regular basis, you might want to go through the file line by line.

Answer 2

Its bad design unless you know the files sizes vs the computer memory that would be avaiable in the running app.

A better solution would be consider memory mapped files. They use themselvses as page fil storage,

Answer 3

On a 32-bit operating system, you'll get at most a contiguous chunk of memory around 550 Megabytes, allowing loading a file of half that size. That goes down hill quickly after your program has been running for a while and the virtual memory address space gets fragmented. 100 Megabytes is about all you can hope for.

This is not an issue on a 64-bit operating system.

Since reading a text file one line at a time is just as fast as reading all lines, this should never be a real problem.