How do you resolve issues with named pipes?

Question

I have a binary program* which takes the contents of a supplied file, processes it, and prints the result on the screen through stdout. For an automation script, I would like to use a named pipe to send data to this program and process the output myself. After trying to get the script to work I realized that there is an issue with the binary program accepting data from the named pipe. To illustrate the problem I have outlined several tests using the unix shell.

1. It is easy to show that the program works by processing an actual data file.

   $ binprog file.txt > output.txt
   

   This will result in output.txt containing the processed information from file.txt.

2. The named pipe (pipe.txt) works as seen by this demonstration.

   $ cat pipe.txt > output.txt
   $ cat file.txt > pipe.txt
   

   This will result in output.txt containing the data from file.txt after it has been sent through the pipe.

3. When the binary program is reading from the named pipe instead of the file, things do not work correctly.

   $ binprog pipe.txt > output.txt
   $ cat file.txt > pipe.txt
   

   In this case output.txt contains no data even after cat and binprog terminate. Using top and ps, I can see binprog "running" and seemingly doing work. Everything executes with no errors.

Why is there no output produced by binprog in this third example?

What are some things I could try to get this working?

[*] The program in question is svm-scale from libsvm. I chose to generalize the examples to keep them clean and simple.

Answer 1

Does binprog also accept input on stdin? If so, this might work for you.

cat pipe.txt | binprog > output.txt
cat file.txt > pipe.txt

Edit: Briefly scanned the manpage for svm-scale. Give this a whirl instead:

cat pipe.txt | svm-scale - > output.txt

Answer 2

Are you sure the program will work with a pipe? If it needs random access to the input file it won't work. The program will get an error whenever it tries to seek in the input file.

If you know the program is designed to work with pipes, and you're using bash, you can use process substitution to avoid having to explicitly create the named pipe.

binprog <(cat file.txt) > output.txt

Answer 3

If binprog is not working well with anything other than a terminal as an input, maybe you need to give it a (pseudo-)terminal (pty) for its input. That is harder to organize, but the expect program is one way of doing that relatively easily. There are discussions of programming with pty's in Rochkind 'Advanced Unix Programming' and in Richards et al 'Advanced Programming in the UNIX Environment'.

Something else to look at is the output of truss or strace or the local equivalent. These programs log all the system calls made by a process. On Solaris, I'd run:

truss -o binprog.truss binprog

interactively, and see what it does. Then I'd try it with i/o redirection, and then with i/o redirection from the named pipe; there may be some significant differences between what it does, or you may see the system call that is hanging. If you see forks in the truss log file, you would need to add a '-f' flag to follow children.