C fork/exec with non-blocking pipe IO

Question

This seems to be a fairly common thing to do, and I've managed to teach myself everything that I need to make it work, except that I now have a single problem, which is defying my troubleshooting.

int nonBlockingPOpen(char *const argv[]){
    int inpipe;
    pid_t pid;
    /* open both ends of pipe nonblockingly */
    pid = fork();

    switch(pid){
        case 0:         /*child*/
            sleep(1); /*child should open after parent has open for reading*/

            /*redirect stdout to opened pipe*/
            int outpipe = open("./fifo", O_WRONLY);
            /*SHOULD BLOCK UNTIL MAIN PROCESS OPENS FOR WRITING*/
            dup2(outpipe, 1);
            fcntl(1, F_SETFL, fcntl(1, F_GETFL) | O_NONBLOCK);

            printf("HELLO WORLD I AM A CHILD PROCESS\n");
            /*This seems to be written to the pipe immediately, blocking or not.*/
            execvp(*argv, argv);
            /*All output from this program, which outputs "one" sleeps for 1 second
             *outputs "two" sleeps for a second, etc, is captured only after the
             *exec'd program exits!
             */
            break;

        default:        /*parent*/
            inpipe = open("./fifo", O_RDONLY | O_NONBLOCK);
            sleep(2);
            /*no need to do anything special here*/
            break;
    }

    return inpipe;
}

Why won't the child process write its stdout to the pipe each time a line is generated? Is there something I'm missing in the way execvp or dup2 work? I'm aware that my approach to all this is a bit strange, but I can't find another way to capture output of closed-source binaries programatically.

Answer 1

I would guess you only get the exec'd program's output after it exits because it does not flush after each message. If so, there is nothing you can do from the outside.

I am not quite sure how this is supposed to relate to the choice between blocking and nonblocking I/O in your question. A non-blocking write may fail completely or partially: instead of blocking the program until room is available in the pipe, the call returns immediately and says that it was not able to write everything it should have. Non-blocking I/O neither makes the buffer larger nor forces output to be flushed, and it may be badly supported by some programs.

You cannot force the binary-only program that you are exec'ing to flush. If you thought that non-blocking I/O was a solution to that problem, sorry, but I'm afraid it is quite orthogonal.

EDIT: Well, if the exec'd program only uses the buffering provided by libc (does not implement its own) and is dynamically linked, you could force it to flush by linking it against a modified libc that flushes every write. This would be a desperate measure. to try only if everything else failed.

Answer 2

Why won't the child process write its stdout to the pipe each time a line is generated?

How do you know that? You do not even try to read the output from the fifo.

N.B. by the file name I presume that you are using the fifo. Or is it a plain file?

And the minor bug in the child: after dup2(), you need to close(outpipe).

fcntl(1, F_SETFL, fcntl(1, F_GETFL) | O_NONBLOCK);

Depending on what program you exec(), you might either lose some output or cause the program to fail since write to stdout now might fail with EWOULDBLOCK.

IIRC fifos has the same buffer size as pipes. Per POSIX minimum is 512 bytes, commonly 4K or 8K.

You probably want to explain why you need that at all. Non-blocking IO has different semantics compared to blocking IO and unless your child process expects that you will run into various problems.

printf("HELLO WORLD I AM A CHILD PROCESS\n");

stdout is buffered, I would have after that fflush(stdout). (Can't find documentation whether exec() on its own would flush stdout or not.)

Is there something I'm missing in the way execvp or dup2 work? I'm aware that my approach to all this is a bit strange, but I can't find another way to capture output of closed-source binaries programatically.

I wouldn't toy with non-blocking IO - and leave it as it is in blocking mode.

And I would use pipe() instead of the fifo. Linux's man pipe has a convenient example with the fork().

Otherwise, that is a pretty normal practice.

Answer 3

When a process is started (via execvp() in your example), the behaviour of standard output depends on whether the output device is a terminal or not. If it is not (and a FIFO is not a terminal), then the output will be fully buffered, rather than line buffered. There is nothing you can do about that; the (Standard) C library does that.

If you really want to make it work line buffered, then you will have to provide the program with a pseudo-terminal as its standard output. That gets into interesting realms - pseudo-terminals or ptys are not all that easy to handle. For the POSIX functions, see:

• grantpt() - grant access to the slave pseudo-terminal device
• posix_openpt() - open a pseudo-terminal device
• ptsname() - get name of the slave pseudo-terminal device
• unlockpt() - unlock a pseudo-terminal master/slave pair

Answer 4

The sleep()s do not guarantee that the parent will open the pipe first - as Dummy00001 says, you should be using a pipe() pipe, not a named pipe. You should also check for execvp() and fork() failing, and you shouldn't be setting the child side to non-blocking - that's a decision for the child process to make.

int nonBlockingPOpen(char *const argv[])
{
    int childpipe[2];
    pid_t pid;

    pipe(childpipe);
    pid = fork();

    if (pid == 0)
    {
        /*child*/

        /*redirect stdout to opened pipe*/
        dup2(childpipe[1], 1);

        /* close leftover pipe file descriptors */
        close(childpipe[0]);
        close(childpipe[1]);

        execvp(*argv, argv);

        /* Only reached if execvp fails */
        perror("execvp");
        exit(1);
    }

    /*parent*/

    /* Close leftover pipe file descriptor */
    close(childpipe[1]);

    /* Check for fork() failing */
    if (pid < 0)
    {
         close(childpipe[0]);
         return -1;
    }

    /* Set file descriptor non-blocking */
    fcntl(childpipe[0], F_SETFL, fcntl(childpipe[0], F_GETFL) | O_NONBLOCK);

    return childpipe[0];
}