StreamWriter .NET - how to get the best performance? C# .Net

Question

What is the recommended approach to get the best performance when we need to create text files bigger than 10MB.

There are multiple sections in the code that need to write stuff to a single file. This means a lot of text lines.

Option #1: (This logic will be called at several times)

1. Create a StreamWriter instance
2. Write some lines (according to some business logic)
3. Close the StreamWriter instance

Option #2:

1. Create a StreamWriter at the begining of the program
2. Write all of the lines from diferent sections of the code.
3. Close the StreamWriter at the very end when nothing else need to be written.

Option #3: Any other?

Remember the output file could be bigger than 10MB.

Answer 1

Holding a single writer open will be more efficient than repeatedly opening and closing it. If this is critical data, however, you should call Flush() after each write to make sure it gets to disk.

Is your program multi-threaded? If so, you may wish to have a producer/consumer queue - have a single thread fetching items to write from the queue and writing them, then other threads can freely put items on the queue.

Are you sure you actually have a performance problem though? 10MB is pretty small these days... on my netbook it still only takes about a second or two to write 10MB (and no, that's not a solid state drive).

Answer 2

Use a StringBuilder to concatenate your text and only open and write once in the file.

Answer 3

In both scenario's 1 and 2 you have to ask yourself whether simultaneous access to the file is required. In this case in scenario 2 StreamWriter is not an option since it isn't synchronized. In scenario 1 you should open each StreamWriter in such a way that it gets an exclusive lock on the file.

Assuming sequential access, I would never go with scenario 2. It requires passing around of your StreamWriter to each section of code that needs it. And who is responsible for closing the writer again. This will quickly become unmaintainable.

Scenario 1 has the disadvantage that you must open a StreamWriter everywhere you need one, which also becomes unmaintainable. Besides, now you have to know in each code section the location of the file.

I would go for a singleton wrapper around the StreamWriter so that you can use it everywhere you like without creating a lot of dependencies on the StreamWriter itself.

Answer 4

Just mix both approaches... Use a buffer that allows u 2 store in memory only as much as you want, once exceded that size your buffer will be written to disk and cleaned.