IOException reading a large file from a UNC path into a byte array using .NET

Question

I am using the following code to attempt to read a large file (280Mb) into a byte array from a UNC path

public void ReadWholeArray(string fileName, byte[] data)
{
    int offset = 0;
    int remaining = data.Length;

    log.Debug("ReadWholeArray");

    FileStream stream = new FileStream(fileName, FileMode.Open, FileAccess.Read);

    while (remaining > 0)
    {
        int read = stream.Read(data, offset, remaining);
        if (read <= 0)
            throw new EndOfStreamException
                (String.Format("End of stream reached with {0} bytes left to read", remaining));
        remaining -= read;
        offset += read;
    }
}

This is blowing up with the following error.

System.IO.IOException: Insufficient system resources exist to complete the requested 

If I run this using a local path it works fine, in my test case the UNC path is actually pointing to the local box.

Any thoughts what is going on here ?

Answer 1

I suspect that something lower down is trying to read into another buffer, and reading all 280MB in one go is failing. There may be more buffers required in the network case than in the local case.

I would read about 64K at a time instead of trying to read the whole lot in one go. That's enough to avoid too much overhead from chunking, but will avoid huge buffers being required.

Personally I tend to just read to the end of the stream rather than assuming that the file size will stay constant. See this question for more information.

Answer 2

It looks like the array wasn't created with sufficient size. How big of an array is allocated prior to being passed in? Or were you assuming the Read function would reallocate the data array as needed? It won't. Edit: Er, maybe not, I just noticed the exception you got. Not sure now.

Answer 3

Also, the code as written needs to put the FileStream into a using block. Failing to dispose of resources is a very possible reason for receiving "Insufficient system resources":

public void ReadWholeArray(string fileName, byte[] data)
{
    int offset = 0;
    int remaining = data.Length;

    log.Debug("ReadWholeArray");

    using(FileStream stream = new FileStream(fileName, FileMode.Open, FileAccess.Read))
    {
        while (remaining > 0)
        {
            int read = stream.Read(data, offset, remaining);
            if (read <= 0)
                throw new EndOfStreamException
                    (String.Format("End of stream reached with {0} bytes left to read", remaining));
            remaining -= read;
            offset += read;
        }
    }
}