I've been trying various methods (popen, pipes + fork/exec, ...) to read a child process' output, all of which are working, but exhibit the same behavior: whenever I try to read the output using read/fread, it only returns when the buffer is completely full, or when the child exits. I'm looking for a behavior that's more like that of sockets: reading any amount of data as soon as some is available.
How do I do that?
I don't think that's possible. The buffering is handled on the child's side, and if it doesn't flush its buffers, there is nothing for you to read. However, a few tools have command line options to control the buffering, e.g. grep --line-buffered.
Generally you don't. In particular, the child process will buffer the stream because it won't see a stream connected to a pipe as being "interactive." Since the buffering is happening inside the child process, about the only way to prevent it is to rewrite the code in the child to prevent it from buffering its standard output (either ever, or when passed a particular switch, or perhaps you can add code to detect when it's connected to a pipe and turn off buffering only in that specific case). That, however, can affect the child's performance if it writes much to standard output (especially if you aren't selective about when you disable buffering).