fclose()/pclose() may block on some file pointers

Question

Calling fclose() here after dup()ing its file descriptor blocks until the child process has ended (presumably because the stream has ended).

FILE *f = popen("./output", "r");
int d = dup(fileno(f));
fclose(f);

However by manually performing the pipe(), fork(), execvp() of the popen(), and then dup()ing the pipe's read file descriptor, closing the original does not block.

int p[2];
pipe(p);
switch (fork()) {
    case 0: {
        char *argv[] = {"./output", NULL};
        close(p[0]);
        dup2(p[1], 1);
        execvp(*argv, argv);
    }
    default: {
        close(p[1]);
        int d = dup(p[0]);
        close(p[0]);
    }
}

Why does this occur, and how can I close the FILE * returned from popen() and use a file descriptor in its place?

Update:

I'm aware that the documentation says to use pclose(), however fclose() blocks as well. Furthermore, I poked around in the glibc code, and pclose() just calls fclose(). The behaviour is the same, whether fclose() or pclose() is used.

Answer 1

The FILE* returned by popen() should be closed by pclose(), not by fclose(). Then, the documentation for pclose() specifies:

The pclose() function waits for the associated process to terminate and returns the exit status of the command as returned by wait4().

So the waiting is one of the two things pclose() does, besides closing the file descriptor. close() does just one thing: it closes the descriptor.

In response to your second question, I think you can use the descriptor returned by fileno(). There's no need to dup() it. After you're done with it, pclose() the original.

Answer 2

http://linux.die.net/man/3/popen

The pclose() function waits for the associated process to terminate and returns the exit status of the command as returned by wait4().

Since pclose() wants to return the exit status, it has to wait for the child to terminate and generate one. Since fclose() calls pclose(), the same applies to fclose() for you.

If you fork and exec and do the rest yourself, you don't end up calling pclose() (directly or indirectly), so there's no waiting at close time. Note however that unless your program is set to ignore SIGCHLD, your process won't terminate (instead it'll go zombie) until the child does. But at least your cost will run to exit first.

Answer 3

Disappointed with the generality of the answers so far (I can RTFM, tyvm), I've investigated this thoroughly, by stepping through and reading the glibc source.

In glibc pclose() directly calls fclose() with no additional effect, so the 2 calls are same. In fact you could use pclose() and fclose() interchangeably. I'm sure this is purely a coincidence in the evolved implementation, and the use of pclose() to close a FILE * returned from popen() is still recommended.

The magic is in popen(). FILE *s in glibc contain a jump table with pointers to appropriate functions to handle such calls as fseek(), fread(), and of relevance fclose(). When calling popen(), a different jump table used than the one used by fopen(). The close member in this jump table points to a special function _IO_new_proc_close, which calls waitpid() on the pid stored in the region pointed to by FILE *.

Here's the relevant call stack in my version of glibc, which I've annotated with notes about what is going on:

// linux waitpid system call interface
#0  0x00f9a422 in __kernel_vsyscall ()
#1  0x00c38513 in __waitpid_nocancel () from /lib/tls/i686/cmov/libc.so.6

// removes fp from a chain of proc files
// and waits for the process of the stored pid to terminate
#2  0x00bff248 in _IO_new_proc_close (fp=0x804b008) at iopopen.c:357

// flushes the stream and calls close in its jump table
#3  0x00c09ff3 in _IO_new_file_close_it (fp=0x804b008) at fileops.c:175

// destroys the FILEs buffers
#4  0x00bfd548 in _IO_new_fclose (fp=0x804b008) at iofclose.c:62

// calls fclose
#5  0x00c017fd in __new_pclose (fp=0x804b008) at pclose.c:43

// calls pclose
#6  0x08048788 in main () at opener.c:34

So the short of it is, using popen(), the returned FILE * must not be closed, even if you dup() its file descriptor, because it will block until the child process terminates. Of course, after this you'll be left with a file descriptor to a pipe which will contain whatever the child process managed to write() to it before terminating.

By not fread()ing with the file pointer returned from popen(), the underlying pipe will not be touched, it's safe to use the file descriptor by fileno(), and finish up by calling pclose().