Slowdown when copying huge (120+GB) files

Question

I have a C# application and am using the FileStream class to read a 120GB file from an *EDIT * isilon storage unit (mapped to z drive) over gigabit LAN. I start by getting 45 megabytes / second read speeds, but at about the 20GB range my read speeds drop dramatically and settles to about 9 megabytes / second. Does anyone have any ideas on what might be causing the slowdown?

Server is Windows Server 2008 Enterprise R2 64 bit, 16 GB RAM, dual quad core CPU and my app is a 64 bit .NET framework 4.0 console application. Here's my code:

        byte[] buffer = new byte[16777216];
        int count;
        long totalBytes = 0;
        FileStream file = File.OpenRead("z:\bigfile.dat");
        while ((count = file.Read(buffer, 0, buffer.Length)) > 0)
        {
            // I track megabyte / second here
            totalBytes += count;
        }

Answer 1

Well it depends on how you manage the stream. Do you read, write to new file, clean the buffer? Because the RAM can obviously not store 120GB into it.

You should give the rest of the code, from what I see right there, there's no way to help you out.

Answer 2

I wonder if the loop itself is holding back GC from collecting the garbage generated in the loop. There is a Microsoft KB article describing the situation for single-threaded console application. I would suggest adding [MTAThread] to the main method, as suggested in the article.

Answer 3

What would be most interesting is to know if CopyFile does the same thing. Without seeing the rest of the code, it's quite hard to say. One not terribly-uncommon thing that happens to folks is that they fill system cache with dirty data, so it screams for a while (just doing a bunch of cached I/O) and then it slows down significantly when the writes start to come. That's when you find out how fast your copy implementation is... The fact that it starts to go south when the size is in the ballpark of the machine's RAM size is a pretty interesting hint...

Answer 4

Turns out I was actually reading over the network. I thought my mapped drive was local, but it was not. I still have the problem when reading over the network, but when I now actually read the file from local disk, speeds are what I expect. Thanks everyone.

EDIT I fixed the problem reading over the network. You can create a FileStream that does not use the windows cache with this code:

FileStream file = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.None, bufferSize, (FileOptions)0x20000000);

The 0x20000000 is a flag that does not have an enumeration in the .NET framework yet, but it basically says to not use the cache.

http://msdn.microsoft.com/en-us/library/cc644950%28v=VS.85%29.aspx

Answer 5

You'd get faster throughput both on the network and on your local machine using async processing...

Look at the BeginRead method...