Most efficient way to create and write multiple 10 KB text files?

Question

We're using the following approach to writing out text files:

public void WriteFile(string filePath, string contents)
{
    if (string.IsNullOrEmpty(filePath)) { throw new ArgumentNullException("filePath"); }
    if (contents == null) { throw new ArgumentNullException("contents"); }

    if (File.Exists(filePath))
    {
        throw new InvalidOperationException(string.Format("The file '{0}' already exists", filePath));
    }

    StreamWriter sw = File.CreateText(filePath); // slow

    sw.Write(contents);
    sw.Close(); // slow
}

We are calling this function a large number of times, so performance is key. The two lines commented slow are where our application is spending most of its time.

The parameter contents contains, on average, about 10 KB of text.

Are there any other approaches in .NET or using the Win32 API that are known to have significantly better performance?

We have already tried the following:

TextWriter tw = new StreamWriter(filePath);

tw.Write(contents);
tw.Close();

But have found the performance to be similar to our initial approach.

Edit

Based on suggestions we also tried:

File.WriteAllText(filePath, contents);

However the performance of this is similar to the other approached listed above.

Answer 1

You could to create another thread to write your file asynchronously in 8k data blocks (internal buffer size).

Answer 2

Sounds like a job for a backgroundworker. Offloads the writing to a new thread, and has events to notify you/the user of progress as well as completion.

Answer 3

What about File.WriteAllText?

Answer 4

The most straight forward way of writing the file would be:

File.WriteAllText(filePath, contents);

If you have performance problem with that, you should look at your requirements. As there really isn't any part of the operation that can be done more efficiently, you should determine if any part of the operation could be skipped or done later.

For example, do you need the file to be completely written and closed when you exit the method? Otherwise you could put the tasks in a queue and let a background thread write file after file at it's own pace.

Answer 5

sw.Write... sw.Flush(); // insert new line sw.Close()...

If you insert the new line above you will probably see your "slow" move to it as the data is written to disk.

Are you running virus scanning software that you can disable? Can you enable write-caching in your battery-backup disk controller?

Answer 6

Forget about you code. The performance problems when working with HDD is the problems of File System, Operation System and of course the hardware quality.

The only thing I can suggest is to redesign your application. For example you can write huge file instead of numerous small files. Just use .tar archiving for all of them, as it does not compress anything but simply stores file streams one by one.