How can I capture rsync progress messages in Perl?

Question

I want to capture all the progress messages emitted by an rsync process from a Perl script. In certain circumstances this isn't working.

Here's a typical rsync command line I use:

rsync -aL --verbose --progress --bwlimit=100 \
  --include-from=/tmp/78hJ2eDCs1 \
  --include '*/' --exclude '*' \
  /srcdir/* \
  hostname:/target/ 2>&1

If I run this within a bash shell, I'll see something like this:

Building file list ...
1600 files...
1700 files...
and so on

If I try the same command within Perl, I get the "Building file list" output OK, but not the status updates. Here's how I test the capture

my $pid = open(OUTPUT, "$cmd |")  or die "Couldn't fork: $!\n";

my $ch;
while(read(OUTPUT, $ch, 1)==1)
{
    print $ch;
}
close(OUTPUT);

My guess is that either rsync senses the output handle isn't a typical console, or is being output in some unusual manner that I'm not capturing. However, what makes it even odder is that if I omit the --include and --exclude filters, I can capture the status messages just fine.

Anyone any clues as to what is going on?

Answer 1

Does perl buffer the output from pipes? If so, you might be able to get it working if after you open the OUTPUT handle, you turn off buffering with OUTPUT->autoflush(1);

Answer 2

you could use Expect.pm, which mimics a PTY, that may give you the output you are looking for.

Failing that, you could try the --stats or --progress options.

Answer 3

Turns out the solution was simple - I just had to unbuffer the IO in my script with $| = 1;

I'm still puzzled by how I observed the problem with some rsync options and not others. Thankyou Paul Tomblin and dsm for giving me ideas.

Answer 4

You might also want to use IPC::Run and it's callbacks for data on stdout/stderr.

Answer 5

Suffering from Buffering? is best explanation about buffering effect I found so far. Well worth the time to read and understand.

Don't forget that buffering is there for a reason so don't just blinding turn it off for every possible problem.