How can I launch a background process in Pylons?

Question

I am trying to write an application that will allow a user to launch a fairly long-running process (5-30 seconds). It should then allow the user to check the output of the process as it is generated. The output will only be needed for the user's current session so nothing needs to be stored long-term. I have two questions regarding how to accomplish this while taking advantage of the Pylons framework:

1. What is the best way to launch a background process such as this with a Pylons controller?

2. What is the best way to get the output of the background process back to the user? (Should I store the output in a database, in session data, etc.?)

Edit: The problem is if I launch a command using subprocess in a controller, the controller waits for the subprocess to finish before continuing, showing the user a blank page that is just loading until the process is complete. I want to be able to redirect the user to a status page immediately after starting the subprocess, allowing it to complete on its own.

Answer 1

I think this has little to do with pylons. I would do it (in whatever framework) in these steps:

• generate some ID for the new job, and add a record in the database.
• create a new process, e.g. through the subprocess module, and pass the ID on the command line (*).
• have the process write its output to /tmp/project/ID
• in pylons, implement URLs of the form /job/ID or /job?id=ID. That will look into the database whether the job is completed or not, and merge the temporary output into the page.

(*) It might be better for the subprocess to create another process immediately, and have the pylons process wait for the first child, so that there will be no zombie processes.

Answer 2

I've handled this problem in the past (long-running process invoked over HTTP) by having my invoked 2nd process daemonize. Your Pylons controller makes a system call to your 2nd process (passing whatever data is needed) and the 2nd process immediately becomes a daemon. This ends the system call and your controller can return.

My web-apps usually then issue AJAX requests to "check-on" the daemon process until it has completed. I've used both tmp files (cPickle works well) and databases to share information between the daemon and the web-app.

Excellent python daemon recipe: http://code.activestate.com/recipes/278731/