Capturing STDOUT and STDERR of an external program *while* it's executing (Ruby)

Question

Scenario:

I have to call an external program from my Ruby script, and this program sends a lot of useful (but cryptic) info to stdout and stderr.

While the program is running, I'd like to parse the lines it sends to stdout and stderr and:

• Remove them if not necessary
• Reformat/replace them if necessary

I tried all the usual tricks (system, exec, popen, popen3, backticks, etc. etc.), but I can only retrieve stdout/stderr after the program is executed, not during its execution.

Any ideas?

Oh, and I'm on Windows :-(

Answer 1

Just an experiment on Linux (sorry, I ain't got Windows, but maybe someone can translate everything to Windowsese - or you are lucky that it works under Cygwin): With the call

ruby producer.rb|ruby consumer.rb

in the command line ('|' symbols a pipe from standard output of producer.rb to standard input of consumer.rb) the output of producer.rb is immediately processed by consumer.rb if the code for both functions is like this:

# producer.rb
1000.times do |i|
  print("%03d " % i)
  STDOUT.flush
  sleep 1
end

# consumer.rb
STDIN.each_byte do |x|
  print('%02x ' % x)
  STDOUT.flush
end

If you comment out any of the two lines with STDOUT.flush, nothing is displayed anymore before the programs are done. The flush in the producer causes its standard output to be forwarded immediately to the consumer. In Linux this is otherwise done whenever a newline is output. If the external program doesn't do either (flush/newlines), you are doomed under Linux.

You asked also about standard error. Under Linux you can either mix standard output and standard error or you can use named pipes. I'm sure there is a way under Linux as long the external program flushes often enough.

Good luck with Windows!

Answer 2

Actually, it was simpler than I thought, this seems to work perfectly:

STDOUT.sync = true # That's all it takes...
IO.popen(command+" 2>&1") do |pipe| # Redirection is performed using operators
  pipe.sync = true
  while str = pipe.gets
    puts "-> "+str # This is synchronous!
  end
end

...and yes, it works on Windows!