C# System.Diagnostics.Process redirecting Standard Out for large amounts of data

Question

I running an exe from a .NET app and trying to redirect standard out to a streamreader. The problem is that when I do

myprocess.exe >> out.txt

out.txt is close to 14mb. When I do the command line version it is very fast but when I run the process from my csharp app it is excruciatingly slow because I believe the default streamreader flushes every 4096 bytes.

Is there a way to change the default stream reader for the Process object?

Answer 1

I haven't tried, but it looks like the asynchronous methods may offer better performance. Instead of using process.StandardOutput, try this method instead:

Process process = Process
    .Start(new ProcessStartInfo("a.exe"){RedirectStandardOutput = true});
if (process != null)
{
    process.OutputDataReceived += ((sender, e) =>
                                   {
                                       string consoleLine = e.Data;
                                       //handle data
                                   });
    process.BeginOutputReadLine();
}

Answer 2

Yes, that's about right. There is a buffer that stores the process output, usually between 1 and 4KB in the common CRT implementations. One small detail: that buffer is located in the process you start, not the .NET program.

Nothing very special needs to happen when you redirect to a file, the CRT directly writes it. But if you redirect to your .NET program then output goes from the buffer into a pipe. Which then takes a thread switch to your program so you can empty the pipe. Back and forth a good 700 times.

Yes, not fast. Easily fixed though, call setvbuf() in the program you are running to increase the stdout and stderr output buffer sizes. Then again, that takes having the source code of that program.

Anticipating a problem with that: maybe you ought to use cmd.exe /c to get the redirection to a file, then read the file.

Answer 3

The Process class exposes the stdout stream directly, so you should be able to read it at whatever pace you like. It's probably best to read it in small chunks and avoid calling ReadToEnd.

For example:

using(StreamReader sr = new StreamReader(myProcess.StandardOutput))
{
  string line;
  while((line = sr.ReadLine()) != null)
  {
    // do something with line
  }
}