Python capture output from wget?

Question

Is it possible to capture output from wget and other command line programs that use curses? Here is what I have right now:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=0)
for line in p.stdout:
    print "a"

This works fine for programs that have simple output, but not for wget and other programs that use curses.

Answer 1

I don't believe that wget is using curses.

Normally when I want to use wget in a script I'd use the -O - option to force it's output to stdout. I suspect you're trying to capture the text that you normally see on your console when you're running it, which would be stderr.

From the command line, outside of Python, just run a command like:

wget -O - http://www.somesite.org/ > /tmp/wget.out 2> /tmp/wget.err

Then look at the two output files. If you see any output from wget on your console/terminal then you are running some different flavor of the command than I've seen.

If, as I suspect, you're actually interested in the stderr messages then you have to choices.

• Change your command to add 2>&1 and add shell=True to your Popen() arguments
• Alternatively add stderr=subprocess.PIPE to your Popen() arguments

The former is handy if you weren't using stdout anyway (assuming your using wget to fetch the data and write it into files). In the latter case you read from the stderr file option to get your data.

BTW: if you really did need to capture curses data ... you could try to use the standard pty module but I wouldn't recommend that. You'd be far better off fetching the pexpect module from:

• [PyPI: pexpect]: http://pypi.python.org/pypi?%3Aaction=search&term=pexpect&submit=search

And don't be scared off by the age or version numbering, it works on Python 2.5 and 2.6 as well as 2.4 and 2.3.