Memory Efficient file append

Question

i have several files whose content need to be merged into a single file. i have the following code that does this ... but it seems rather inefficient in terms of memory usage ... would you suggest a better way to do it ?

the Util.MoveFile function simply accounts for moving files across volumes

   private void Compose(string[] files)
   {
       string inFile = "";
       string outFile = "c:\final.txt";

       using (FileStream fsOut = new FileStream(outFile + ".tmp", FileMode.Create))
       {
           foreach (string inFile in files)
           {
               if (!File.Exists(inFile))
               {
                   continue;
               }

               byte[] bytes;
               using (FileStream fsIn = new FileStream(inFile, FileMode.Open))
               {
                   bytes = new byte[fsIn.Length];
                   fsIn.Read(bytes, 0, bytes.Length);
               }

               //using (StreamReader sr = new StreamReader(inFile))
               //{
               //    text = sr.ReadToEnd();
               //}

               // write the segment to final file
               fsOut.Write(bytes, 0, bytes.Length);

               File.Delete(inFile);
           }
       }

       Util.MoveFile(outFile + ".tmp", outFile);

}

Answer 1

Sometimes its just better to call shell function than to reimplement functionality. As Alan says you can use CAT on unix systems or perhaps on windows you can use the built in command processor

copy file1+file2+file3 concated_file

Answer 2

You can use smaller fixed-size buffers like so:

byte[] bytes = new byte[8192]; // adjust this as needed
int bytesRead;
do {
    bytesRead = fsIn.Read(bytes, 0, bytes.Length);
    fsOut.Write(bytes, 0, bytesRead);
} while (bytesRead > 0);

This is pretty self explanatory except for during the last block so basically what's happening is I'm passing an 8K byte array to the Read method which returns the number of bytes it actually read. So on the Write call, I am passing that value which is somewhere between 0 and 8192. In other words, on the last block, even though I am passing a byte array of 8192 bytes, bytesRead might only be 10 in which case only the first 10 bytes need to be written.

EDIT

I edited my answer to do this in a slightly different way. Instead of using the input file's position to determine when to break out of the loop, I am checking to see if bytesRead is greater than zero. This method works with any kind of stream to stream copy including streams that don't have a fixed or known length.