Force another program's standard output to be unbuffered using Python

Question

A python script is controlling an external application on Linux, passing in input via a pipe to the external applications stdin, and reading output via a pipe from the external applications stdout.

The problem is that writes to pipes are buffered by block, and not by line, and therefore delays occur before the controlling script receives data output by, for example, printf in the external application.

The external application cannot be altered to add explicit fflush(0) calls.

How can the pty module of the python standard library be used with the subprocess module to achieve this?

Answer 1

Try running the Python interpreter with the -u argument:

python -u myscript.py

This forces Python to use unbuffered stdin/stdout which may help you.

Answer 2

I don't think it's possible. If the source application doesn't flush its outgoing buffer, the data will not reach outside that process until the buffer overflows and a flush is forced.

Notice how a well-established command such as file has an option (-n) that causes it to flush its output explicitly. This is required when using file in the mode where it reads input file names from a pipe, and prints the detected type. Since in this mode, the file program doesn't quit when done, the output would otherwise not appear.

Consider this at a lower level: the output buffering simply means that doing write() on a buffered stream copies the data into an in-memory buffer, until the buffer fills up or (typically) until a linefeed is found. Then, the part of the buffer up to the overflow or linefeed is written write()n to the underlying system-level file descriptor (which could be a file, a pipe, a socket, ...).

I don't understand how you're going to convince that program to flush its buffer, from the outside.

Answer 3

Doing this is possible, but the only solution I can think of is fairly convoluted, non-portable, and probably fraught with problematic details. You can use LD_PRELOAD to cause the external application to load a dynamic library which contains a constructor that invokes setvbuf to unbuffer stdout. You will probably also want to wrap setvbuf in the library to prevent the application from explicitly buffering its own stdout. And you'll want to wrap fwrite and printf so that they flush on each call. Writing the .so to be preloaded will take you outside of python.

Answer 4

Its worth noting that some programs only buffer their output when they think it's not going to a "real user" (ie, a tty). When they detect that their output is being read by another program, they buffer.

The emulation of a tty is one of the things that Expect does in automating other processes.

There is a pure Python implementation of Expect, but I don't know how well it handles tty emulation.

Answer 5

You can use PTYs to solve this by:

• Creating a pty master/slave pair;
• Connecting the child process's stdin, stdout and stderr to the pty slave device;
• Reading from and writing to the pty master in the parent.