How to execute Python script from C code using EXECL?

Question

I would like to know how I can execute a Python (or Lua etc) script from my C code using execl (or similar)?

The following is some "parent / child" code showing how I am sending a STREAM of data to the child using PIPES. The code may not be perfect but you get the idea. Note the execl at the bottom:

#include <sys/types.h> 
#include <stdio.h>  
#include <stdlib.h>  
#include <string.h>  
#include <unistd.h>  

#define STDIN_FILENO    0       /* Standard input.  */
#define STDOUT_FILENO   1       /* Standard output.  */
#define STDERR_FILENO   2       /* Standard error output.  */
#define MAXLINE 4096

int main(void){
 int  n, parent_child_pipe[2], child_parent_pipe[2];
 pid_t pid;
 char line[MAXLINE];

 if (pipe(parent_child_pipe) < 0 || pipe(child_parent_pipe) < 0)
  puts("Error creating pipes...\n");

 if ( (pid = fork()) < 0)
  puts("Error forking...\n");
 else if (pid > 0) { /* PARENT */
  close(parent_child_pipe[0]); 
  close(child_parent_pipe[1]);
  while (fgets(line, MAXLINE, stdin) != NULL) {
   n = strlen(line);
   if (write(parent_child_pipe[1], line, n) != n)
    puts("write error to pipe...\n");
   if ( (n = read(child_parent_pipe[0], line, MAXLINE)) < 0)
    puts("read error from pipe...\n");
   if (n == 0) {
    puts("child closed pipe...\n");
    break;
   }
   line[n] = 0; /* null terminate */
   if (fputs(line, stdout) == EOF)
    puts("fputs error...\n");
  }
  if (ferror(stdin))
   puts("fgets error on stdin...\n");
  exit(0);

 } else {  /* CHILD */
  close(parent_child_pipe[1]);
  close(child_parent_pipe[0]);
  if (parent_child_pipe[0] != STDIN_FILENO) {
   if (dup2(parent_child_pipe[0], STDIN_FILENO) != STDIN_FILENO)
    puts("dup2 error to stdin...\n");
   close(parent_child_pipe[0]);
  }
  if (child_parent_pipe[1] != STDOUT_FILENO) {
   if (dup2(child_parent_pipe[1], STDOUT_FILENO) != STDOUT_FILENO)
    puts("dup2 error to stdout...\n");
   close(child_parent_pipe[1]);
  }
  **if (execl("./child", "child", (char *) 0) < 0)**
   puts("execl error...\n");
 }
}

Now the "child" program above is written in C and simply receives the stream via STDIN, manipulates the stream, and sends it back out using STDOUT.

Something like:

#include <stdio.h>  
#include <stdlib.h>  
#include <string.h>  
#include <unistd.h>  

#define STDIN_FILENO    0       /* Standard input.  */
#define STDOUT_FILENO   1       /* Standard output.  */
#define STDERR_FILENO   2       /* Standard error output.  */
#define MAXLINE 4096

int main(void){
 int  n;
 char line[MAXLINE];

 while ( (n = read(STDIN_FILENO, line, MAXLINE)) > 0) {
  line[n] = 0;  /* null terminate */  
  n = strlen(line);
  if (write(STDOUT_FILENO, line, n) != n)
   puts("write error");
 }
 exit(0);
}

So that is working fine, but now I want to be able to write the CHILD in any scripting language like Python / Lua etc. How can I do this? I have tried stuff like:

if (execl("path to python", "test.py", (char *) 0) < 0)

But it just seems to HANG waiting to receive input?

Can someone please help me on this? I assume PIPES can talk to anything like Lua / Python that can read from STDIN and send back to STDOUT?

UPDATE:

I have made some small changes now, here is the Python file: "NullFilter.py" that simply ECHOS what it is sent:

#!/usr/bin/python
import sys

class NullFilter:

      def execute(self): 
            #Read data from STDIN...
            data = sys.stdin.read() 
            #Write data to STDOUT...
            sys.stdout.write(data) 
            exit(0) 
if __name__ == '__main__':
      nf = NullFilter()
      nf.execute()

And now the C code calls it using:

...
if (execl("/usr/bin/python","./NullFilter.py","./NullFilter.py",NULL, (char *) 0) < 0)
puts("execl error...\n");
...

When I run it now I can enter text into STDIN, but have to hit CRTL-C to see what has happened: Here is the result:

debian@debian:~/Desktop/pipe example$ ./parent
hello
hello again
^CTraceback (most recent call last):
  File "./NullFilter.py", line 17, in <module>

nf.execute()
  File "./NullFilter.py", line 10, in execute
debian@debian:~/Desktop/pipe example$     data = sys.stdin.read() 
KeyboardInterrupt

debian@debian:~/Desktop/pipe example$ 

UPDATE 2:

OK, so I have been playing around a little, and simply changed the Python code from "sys.stdin.read()" to "sys.stdin.readline()" and it seems to work to a certain degree, BUT NOT PERFECT at all...

#!/usr/bin/python
import sys

class NullFilter:

      def execute(self): 
            #Read data from STDIN...
            data = ""             
            for line in sys.stdin.readline():
                data = data + line 
            #Write data to STDOUT...
            sys.stdout.write(data) 
            exit(0) 
if __name__ == '__main__':
      nf = NullFilter()
      nf.execute()

This is a workaround BUT not perfect at all. Any other ideas how I can get the STDIN to read the stream UNBUFFERED? I have looked at the SELECT module in Python:

http://docs.python.org/library/select.html

I have also tried passing "-u" to Python to make it "unbuffered" but still no luck ;-(

But surely this cannot be so difficult?

Lynton

Answer 1

I believe you can achieve what you want to do by starting the python file with the shebang line like this:

#!/usr/bin/python

and ensuring it is executable, then using the script as the executable field in execl()

See http://linux.about.com/od/commands/l/blcmdl2_execve.htm:

Filename must be either a binary executable, or a script starting with a line of the form "#! interpreter [arg]". In the latter case, the interpreter must be a valid pathname for an executable which is not itself a script, which will be invoked as interpreter [arg] filename.

In the case of passing arguments (also theoretically valid), it doesn't look like python is actually executing the script and finishing as it should - rather it is opening an interactive terminal which is why it hangs - it wants you to communicate via stdin. Try the absolute path to "test.py" and see what happens, as it sounds like python is unable to find it.

There is, actually, another method. There is no reason you can't pass the script line-by-line to the interpreter via pipes and run it that way.

Answer 2

Did You try a flush on the child ?

UPDATE:

I tried with your C code and with these 2 scripts (perl and python) and their behavior is exactly like the C child on my machine

#!/usr/bin/perl -w
use IO::Handle; 

while (<>)
{
        print;
        flush STDOUT;
}

And

#!/usr/bin/python
import sys

class NullFilter:
        def execute(self): 
                while True:
                        line=sys.stdin.readline()
                        if not line: break
                        sys.stdout.write(line)
                        sys.stdout.flush()
                exit(0) 
if __name__ == '__main__':
        nf = NullFilter()
        nf.execute()

Answer 3

Hi there,

After much playing around and trying to flush everything I realised I needed to CLOSE the STDOUT end of the PIPE going from the parent to the child (after writing to the pipe of course)...

So the code is now:

#include <sys/types.h>  
#include <stdio.h>   
#include <stdlib.h>   
#include <string.h>   
#include <unistd.h>   

#define STDIN_FILENO    0       /* Standard input.  */ 
#define STDOUT_FILENO   1       /* Standard output.  */ 
#define STDERR_FILENO   2       /* Standard error output.  */ 
#define MAXLINE 4096 

int main(void){ 
 int  n, parent_child_pipe[2], child_parent_pipe[2]; 
 pid_t pid; 
 char line[MAXLINE]; 
 int rv;

 if (pipe(parent_child_pipe) < 0 || pipe(child_parent_pipe) < 0) 
  puts("Error creating pipes...\n"); 

 if ( (pid = fork()) < 0) 
  puts("Error forking...\n"); 
 else if (pid > 0) { /* PARENT */ 
  close(parent_child_pipe[0]);  
  close(child_parent_pipe[1]); 
  while (fgets(line, MAXLINE, stdin) != NULL) { 
   n = strlen(line); 
   if (write(parent_child_pipe[1], line, n) != n) 
    puts("write error to pipe...\n"); 
   close(parent_child_pipe[1]);
   wait(&rv); 
   if ( (n = read(child_parent_pipe[0], line, MAXLINE)) < 0) 
    puts("read error from pipe...\n"); 
   if (n == 0) { 
    puts("child closed pipe...\n"); 
    break; 
   } 
   line[n] = 0; /* null terminate */ 
   if (fputs(line, stdout) == EOF) 
    puts("fputs error...\n"); 
  } 
  if (ferror(stdin)) 
   puts("fgets error on stdin...\n"); 
  exit(0); 

 } else {  /* CHILD */ 
  close(parent_child_pipe[1]); 
  close(child_parent_pipe[0]); 
  if (parent_child_pipe[0] != STDIN_FILENO) { 
   if (dup2(parent_child_pipe[0], STDIN_FILENO) != STDIN_FILENO) 
    puts("dup2 error to stdin...\n"); 
   close(parent_child_pipe[0]); 
  } 
  if (child_parent_pipe[1] != STDOUT_FILENO) { 
   if (dup2(child_parent_pipe[1], STDOUT_FILENO) != STDOUT_FILENO) 
    puts("dup2 error to stdout...\n"); 
   close(child_parent_pipe[1]); 
  } 
  if (execl("./NullFilter.py", "./NullFilter.py", (char *) 0) < 0)
   puts("execl error...\n"); 
 } 
} 

You can see the "close(parent_child_pipe[1]);" just after writing to the PIPE above, that is the crucial piece I had to do. That will force the stream to be flushed to the Python script, Lua script , C code etc....

In the above you will see I am executing a Python script "NullFilter.py"....

NOTE: If you run the code above it will work for one iteration of input / output as the Python script closes the pipe after the first test, but the essentials are there for you to build on...

Thanks for all the help though, I have learnt a lot from this exercise ;-)

Lynton