How can I perform a ping or traceroute in python, accessing the output as it is produced?

Question

Earlier, I asked this question:

How can I perform a ping or traceroute using native python?

However because python is not running as root it doens't have the ability to open the raw ICMP sockets needed to perform the ping/traceroute in native python.

This brings me back to using the system's ping/traceroute shell commands. This question has a couple examples using the subprocess module which seem to work well:

Ping a site in Python?

I still have one more requirement though: I need to be able to access the output as it is produced (eg. for a long running traceroute.)

The examples above all run the shell command and then only give you access to the complete output once the command has completed. Is there a way to access the command output as it is produced?

Edit: Based on Alex Martelli's answer, here's what worked:

import pexpect

child = pexpect.spawn('ping -c 5 www.google.com')

while 1:
        line = child.readline()
        if not line: break
        print line,

Answer 1

pexpect is what I'd reach for, "by default", for any requirement such as yours -- there are other similar modules, but pexpect is almost invariably the richest, most stable, and most mature one. The one case where I'd bother looking for alternatives would be if I had to run correctly under Windows too (where ping and traceroute may have their own problems anyway) -- let us know if that's the case for you, and we'll see what can be arranged!-)

Answer 2

You should read the documentation for the subprocess module, it describes how to run an external process and access its output in real time.

Basically, you do

from subprocess import Popen, PIPE
p = Popen(['tracert', host], stdout=PIPE)
while True:
    line = p.stdout.readline()
    if not line:
        break
    # Do stuff with line

Actually, the answers in the SO question you linked to are very close to what you need. Corey Goldberg's answer uses a pipe and readline, but since it runs ping with -n 1 it doesn't last long enough to make a difference.

Answer 3

You can create a tty pair for the subprocess and run inside of that. According to the C standard (C99 7.19.3) the only time stdout is line buffered (as opposed to fully buffered which is what you say you don't want) is when it's a terminal. (or the child called setvbuf() obviously).

Check out os.openpty().

Untested code:

master, slave = os.openpty()
pid = os.fork()
if pid == 0:
    os.close(master)
    os.dup2(slave, 0)
    os.dup2(slave, 1)
    os.dup2(slave, 2)
    os.execv("/usr/sbin/traceroute", ("traceroute","4.2.2.1"))
    # FIXME: log error somewhere
    os.exit(1)
os.close(slave)
while True:
    d = os.read(master)
    if len(d) == 0:
        break
    print d
os.waitpid(pid, 0)

Note that having the child process (just after fork()) call setvbuf() will not work, since setvbuf() is a libc function and not a syscall. It just changes the state of the current process output, which will be overwritten on the exec call when the new binary in loaded.