Can I capture stdout/stderr separately and maintain original order?

Question

I've written a Windows application using the native win32 API. My app will launch other processes and capture the output and highlight stderr output in red.

In order to accomplish this I create a separate pipe for stdout and stderr and use them in the STARTUPINFO structure when calling CreateProcess. I then launch a separate thread for each stdout/stderr handle that reads from the pipe and logs the output to a window.

This works fine in most cases. The problem I am having is that if the child process logs to stderr and stdout in quick succession, my app will sometimes display the output in the incorrect order. I'm assuming this is due to using two threads to read from each handle.

Is it possible to capture stdout and stderr in the original order they were written to, while being able to distinguish between the two?

Answer 1

Not really, you would think so but std_out is at the control of the system designers - exactly how and when std_out gets written is subject to system scheduler, which by my testing is subordinated to issues that are not as documented.

I was writing some stuff one day and did some work on one of the devices on the system while I had the code open in the editor and discovered that the system was giving real-time priority to the driver, leaving my carefully-crafted c-code somewhere about one tenth as important as the proprietary code.

Re-inverting that so that you get sequential ordering of the writes is gonna be challenging to say the least.

Answer 2

You can redirect stderr to stdout:

command_name 2>&1

This is possible in C using pipes, as I recall.

UPDATE: Oh, sorry -- missed the part about being able to distinguish between the two. I know TextMate did it somehow using kinda user visible code... Haven't looked for a while, but I'll give it a peek. But after some further thought, could you use something like Open3 in Ruby? You'd have to watch both STDOUT and STDERR at the same time, but really no one should expect a certain ordering of output regarding these two.

UPDATE 2: Example of what I meant in Ruby:

require 'open3'

Open3.popen3('ruby print3.rb') do |stdin, stdout, stderr|
  loop do
    puts stdout.gets
    puts stderr.gets
  end
end

...where print3.rb is just:

loop do
  $stdout.puts 'hello from stdout'
  $stderr.puts 'hello from stderr'
end

Instead of throwing the output straight to puts, you could send a message to an observer which would print it out in your program. Sorry, I don't have Windows on this machine (or any immediately available), but I hope this illustrates the concept.

Answer 3

I'm pretty sure it can't be done, short of writing the spawned program to write in packets and add a time-stamp to each. Without that, you can normally plan on buffering happening in the standard library of the child process, so by the time they're even being transmitted through the pipe to the parent, there's a good chance that they're already out of order.

Answer 4

I'm pretty sure that even if you don't separate them at all, you're still not guaranteed that they'll interchange one another in the correct order.

Answer 5

In most implementations of stdout and stderr that I've seen, stdout is buffered and stderr is not. Basically what this means is that you aren't guaranteed they're going to be in order even when running the program on straight command line.

http://en.wikipedia.org/wiki/Stderr#Standard%5Ferror%5F.28stderr.29

The short answer: You cannot ensure that you read the lines in the same order that they appear on cmd.exe because the order they appear on cmd.exe is not guaranteed.

Answer 6

Since the intent is to annotate the output os an existing program, any possible interleaving of the two streams must be correct. The original developer will have placed appropriate flush() calls to ensure any mandatory ordering is honoured.

As previously explained, record each fragment that is written with a time stamp, and use this to recover the sequence actually seen by the output devices.