wget not behaving via IPC::Open3 vs bash

Question

I'm trying to stream a file from a remote website to a local command and am running into some problems when trying to detect errors.

The code looks something like this:

use IPC::Open3;

my @cmd = ('wget','-O','-','http://10.10.1.72/index.php');#any website will do here

my ($wget_pid,$wget_in,$wget_out,$wget_err);
if (!($wget_pid = open3($wget_in,$wget_out,$wget_err,@cmd))){
    print STDERR "failed to run open3\n";
    exit(1)
}
close($wget_in);
my @wget_outs = <$wget_out>;
my @wget_errs = <$wget_err>;
print STDERR "wget stderr: ".join('',@wget_errs);
#page and errors outputted on the next line, seems wrong
print STDERR "wget stdout: ".join('',@wget_outs);

#clean up after this, not shown is running the filtering command, closing and waitpid'ing

When I run that wget command directly from the command-line and redirect stderr to a file, something sane happens - the stdout will be the downloaded page, the stderr will contain the info about opening the given page.

wget -O - http://10.10.1.72/index.php 2> stderr_test_file

When I run wget via open3, I'm getting both the page and the info mixed together in stdout. What I expect is the loaded page in one stream and STDERR from wget in another.

I can see I've simplified the code to the point where it's not clear why I want to use open3, but the general plan is that I wanted to stream stdout to another filtering program as I received it, and then at the end I was going to read the stderr from both wget and the filtering program to determine what, if anything went wrong.

Other important things:
• I was trying to avoid writing the wget'd data to a file, then filtering that file to another file, then reading the output.
• It's key that I be able to see what went wrong, not just reading $? >> 8 (i.e. I have to tell the user, hey, that IP address is wrong, or isn't the right kind of website, or whatever).
• Finally, I'm choosing system/open3/exec over other perl-isms (i.e. backticks) because some of the input is provided by untrustworthy users.

Answer 1

You are passing an undefined value as the error handle argument to open3, and as IPC::Open3 says:

If CHLD_ERR is false, or the same file descriptor as CHLD_OUT, then STDOUT and STDERR of the child are on the same filehandle (this means that an autovivified lexical cannot be used for the STDERR filehandle, see SYNOPSIS) ...

The workaround is to initialize $wget_err to something before calling open3:

my ($wget_pid, $wget_in, $wget_out, $wget_err);
use Symbol qw(gensym);
$wget_err = gensym();
if (!$wget_pid = open3( ... ) ) { ...