How can I run a long background process from a Perl CGI program?

Question

I have a problem with a long running CGI and the timeout error:

Timeout waiting for output from CGI script

The client side is a form programmed in jQuery. The user input some data and receives a message that the analyses has been launch. The user don't expect to see receive more messages except an email with a link when the data has been analyzed. So, at this point, the connection with the client is closed, right?

In the server side, a Perl CGI script gets the data and executes some C programs (using Perl's system) to analyze them. This process can take from a few seconds to hours depending on the inputed data.

Then the same CGI program parses the results and sends an email to the user with a link to the results webpage.

Since for some data, the CGI can be running for hours I am getting the error message.

I am assuming that increasing ScriptTimeout is a bad idea. I am not even sure that mod_cgi is installed.

What can I do to avoid this error?

Server: Apache2 running in Mac OS X.

Answer 1

The CGI should not be doing this work itself. It should instead simply gather the user input and finish immediately, and then dispatch a separate program to do the work offline. A common solution is to use a worker queue to store these requests from users, and the separate program listens to this queue and performs work as requested.

Edit: Typically, there would be a daemon running all the time that listens to the queue (e.g. at my $work I have a worker daemon that uses Beanstalk::Client and beanstalkd for its job queue), but if you have jobs being added only infrequently, then a cron job is a good first implementation.

As an alternate solution, you can fork your CGI and call exec in the child to start your worker program:

# there is work to be done; dispatch the worker script in a child process.
my $pid = fork;
exec "/path/to/worker/script.pl", $arg1, $arg2 if not $pid;

# parent CGI is still alive; return an acknowledgement to the user and return.

Answer 2

In these situations, I call an external program whose only job is to fork off the process. The first process can dissociate from its child and return immediately while the real work keeps going.

You might also see How can I fork a Perl CGI program to hive off long-running tasks?