Dealing with external processes

Question

I've been working on a gui app that needs to manage external processes. Working with external processes leads to a lot of issues that can make a programmer's life difficult. I feel like maintenence on this app is taking an unacceptably long time. I've been trying to list the things that make working with external processes difficult so that I can come up with ways of mitigating the pain. This kind of turned into a rant which I thought I'd post here in order to get some feedback and to provide some guidance to anybody thinking about sailing into these very murky waters. Here's what I've got so far:

1. Output from the child can get mixed up with output from the parent. This can make both outputs misleading and hard to read. It can be hard to tell what came from where. It becomes harder to figure out what's going on when things are asynchronous. Here's a contrived example:

   import textwrap, os, time
   from subprocess import Popen
   test_path = 'test_file.py'
   
   
   with open(test_path, 'w') as file:
       file.write(textwrap.dedent('''
           import time
           for i in range(3):
               print 'Hello %i' % i
               time.sleep(1)'''))
   
   
   proc = Popen('python -B "%s"' % test_path)
   
   
   for i in range(3):
       print 'Hello %i' % i
       time.sleep(1)
   
   
   os.remove(test_path)
   

   Output:

   Hello 0
   Hello 0
   Hello 1
   Hello 1
   Hello 2
   Hello 2
   

   I guess I could have the child process write its output to a file. But it can be annoying to have to open up a file every time I want to see the result of a print statement.

   If I have code for the child process I could add a label, something like print 'child: Hello %i', but it can be annoying to do that for every print. And it adds some noise to the output. And of course I can't do it if I don't have access to the code.

   I could manually manage the process output. But then you open up a huge can of worms with threads and polling and stuff like that.

   A simple solution is to treat processes like synchronous functions, that is, no further code executes until the process completes. In other words, make the process block. But that doesn't work if you're building a gui app. Which brings me to the next problem...

2. Blocking processes cause the gui to become unresponsive.

   import textwrap, sys, os
   from subprocess import Popen
   
   
   from PyQt4.QtGui import *
   from PyQt4.QtCore import *
   
   
   test_path = 'test_file.py'
   with open(test_path, 'w') as file:
       file.write(textwrap.dedent('''
           import time
           for i in range(3):
               print 'Hello %i' % i
               time.sleep(1)'''))
   
   
   app = QApplication(sys.argv)
   button = QPushButton('Launch process')
   def launch_proc():
       # Can't move the window until process completes
       proc = Popen('python -B "%s"' % test_path)
       proc.communicate()
   button.connect(button, SIGNAL('clicked()'), launch_proc)
   button.show()
   app.exec_() 
   os.remove(test_path)
   

   Qt provides a process wrapper of its own called QProcess which can help with this. You can connect functions to signals to capture output relatively easily. This is what I'm currently using. But I'm finding that all these signals behave suspiciously like goto statements and can lead to spaghetti code. I think I want to get sort-of blocking behavior by having the 'finished' signal from QProcess call a function containing all the code that comes after the process call. I think that should work but I'm still a bit fuzzy on the details...

3. Stack traces get interrupted when you go from the child process back to the parent process. If a normal function screws up, you get a nice complete stack trace with filenames and line numbers. If a subprocess screws up, you'll be lucky if you get any output at all. You end up having to do a lot more detective work everytime something goes wrong.

4. Speaking of which, output has a way of disappearing when dealing external processes. Like if you run something via the windows 'cmd' command, the console will pop up, execute the code, and then disappear before you have a chance to see the output. You have to pass the /k flag to make it stick around. Similar issues seem to crop up all the time.

   I suppose both problems 3 and 4 have the same root cause: no exception handling. Exception handling is meant to be used with functions, it doesn't work with processes. Maybe there's some way to get something like exception handling for processes? I guess that's what stderr is for? But dealing with two different streams can be annoying in itself. Maybe I should look into this more...

5. Processes can hang and stick around in the background without you realizing it. So you end up yelling at your computer cuz it's going so slow until you finally bring up your task manager and see 30 instances of the same process hanging out in the background.

   Also, hanging background processes can interefere with other instances of the process in various fun ways, such as causing permissions errors by holding a handle to a file or someting like that.

   It seems like an easy solution to this would be to have the parent process kill the child process on exit if the child process didn't close itself. But if the parent process crashes, cleanup code might not get called and the child can be left hanging.

   Also, if the parent waits for the child to complete, and the child is in an infinite loop or something, you can end up with two hanging processes.

   This problem can tie in to problem 2 for extra fun, causing your gui to stop responding entirely and force you to kill everything with the task manager.

6. F***ing quotes

   Parameters often need to be passed to processes. This is a headache in itself. Especially if you're dealing with file paths. Say... 'C:/My Documents/whatever/'. If you don't have quotes, the string will often be split at the space and interpreted as two arguments. If you need nested quotes you can use ' and ". But if you need to use more than two layers of quotes, you have to do some nasty escaping, for example: "cmd /k 'python \'path 1\' \'path 2\''".

   A good solution to this problem is passing parameters as a list rather than as a single string. Subprocess allows you to do this.

7. Can't easily return data from a subprocess.

   You can use stdout of course. But what if you want to throw a print in there for debugging purposes? That's gonna screw up the parent if it's expecting output formatted a certain way. In functions you can print one string and return another and everything works just fine.

8. Obscure command-line flags and a crappy terminal based help system.

   These are problems I often run into when using os level apps. Like the /k flag I mentioned, for holding a cmd window open, who's idea was that? Unix apps don't tend to be much friendlier in this regard. Hopefully you can use google or StackOverflow to find the answer you need. But if not, you've got a lot of boring reading and frusterating trial and error to do.

9. External factors.

   This one's kind of fuzzy. But when you leave the relatively sheltered harbor of your own scripts to deal with external processes you find yourself having to deal with the "outside world" to a much greater extent. And that's a scary place. All sorts of things can go wrong. Just to give a random example: the cwd in which a process is run can modify it's behavior.

There are probably other issues, but those are the ones I've written down so far. Any other snags you'd like to add? Any suggestions for dealing with these problems?

Answer 1

Check out the subprocess module. It should help with output separation. I don't see any way around either separate output streams or some kind of output tagging in a single stream.

The hanging process problem is difficult as well. The only solution I have been able to make is to put a timer on the external process, and kill it if it does not return in the allotted time. Crude, nasty, and if anyone else has a good solution, I would love to hear it so I can use it too.

One thing you could do to help deal with the problem of completely un-managed shutdown is to keep a directory of pid files. Whenever you kick off an external process, write a file into your pid file directory with a name that is the pid for the process. Erase the pid file when you know the process has exited cleanly. You can use the stuff in the pid directory to help cleanup on crashes or re-starts.

This may not provide any satisfying or useful answers, but maybe it's a start.