Getting another program's output as input on the fly

Question

I've two programs I'm using in this way:

$ c_program | python_program.py

c_program prints something using printf() and python_program.py reads using sys.stdin.readline()

I'd like to make the python_program.py process c_program's output as it prints, immediately, so that it can print its own current output. Unfortunately python_program.py gets its input only after c_program ends.

How can I solve this?

Answer 1

All the Unix shells (that I know of) implement shell pipelines via something else than a pty (typically, they use Unix pipes!-); therefore, the C/C++ runtime library in cpp_program will KNOW its output is NOT a terminal, and therefore it WILL buffer the output (in chunks of a few KB at a time). Unless you write your own shell (or semiquasimaybeshelloid) that implements pipelines via pyt's, I believe there is no way to do what you require using pipeline notation.

The "shelloid" thing in question might be written in Python (or in C, or Tcl, or...), using the pty module of the standard library or higher-level abstraction based on it such as pexpect, and the fact that the two programs to be connected via a "pty-based pipeline" are written in C++ and Python is pretty irrelevant. The key idea is to trick the program to the left of the pipe into believing its stdout is a terminal (that's why a pty must be at the root of the trick) to fool its runtime library into NOT buffering output. Once you have written such a shelloid, you'd call it with some syntax such as:

$ shelloid 'cpp_program | python_program.py'

Of course it would be easier to provide a "point solution" by writing python_program in the knowledge that it must spawn cpp_program as a sub-process AND trick it into believing its stdout is a terminal (i.e., python_program would then directly use pexpect, for example). But if you have a million of such situations where you want to defeat the normal buffering performed by the system-provided C runtime library, or many cases in which you want to reuse existing filters, etc, writing shelloid might actually be preferable.

Answer 2

You may want to try flushing the stdout stream in the cpp program.

Answer 3

What you need is for your C program to call fflush(stdout) after every line. For example, with the GNU grep tool, you can invoke the option '--line-buffered', which causes this behavior. See fflush.

Answer 4

ok this maybe sound stupid but it might work:

output your pgm to a file

$ c_program >> ./out.log

develop a python program that read from tail command

import os

tailoutput = os.popen("tail -n 0 -f ./out.log")

try:
    while 1:
        line = tailoutput.readline()
        if len(line) == 0:
            break

        #do the rest of your things here
        print line

except KeyboardInterrupt:
        print "Quitting \n"

Answer 5

Just set stdout to be line buffered at the beginning of your C program (before performing any output), like this:

#include <stdio.h>
setvbuf(stdout, NULL, _IOLBF, 0);

or

#include <stdio.h>
setlinebuf(stdout);

Either one will work on Linux, but setvbuf is part of the C standard so it will work on more systems.

By default stdout will be block buffered for a pipe or file, or line buffered for a terminal. Since stdout is a pipe in this case, the default will be block buffered. If it is block buffered then the buffer will be flushed when it is full, or when you call fflush(stdout). If it is line buffered then it will be flushed automatically after each line.

Answer 6

If you can modify your C program, you've already received your answer but i thought i'd include a solution for those that can't/won't modify code.

expect has an example script called unbuffer that will do the trick.