Running background processes after a web request

Question

I'm interested in kicking off processes after a web request, or possibly forking a new process after the initial thread is finished.

I would prefer not to use a cron, because of the nature of the the jobs I'll be running and how often they need to be run, waiting a minute to refresh is not an option.

I'm considering a few ways of doing this:

1) Calling a page in javascript that kicks off the process and immediately returns, then runs tasks after, for instance ajax('/run_jobs.php?job=123').... you get the idea

2) Forking a new thread after a thread has finished; ie output_page(); new thread(); run_job(123); exit();

Anyone have any ideas on the topic or have experience with this.

Answer 1

These days the standard way of handling this kind of thing is a Messaging Queue system. At the end of your request you send a message into the queue. Then, there's some kind of external process running (multiple crons, for example) that's constantly reading messages back out of the queue and running the jobs that need to be run

Zend Queue is a popular choice for PHP.

As for your proposed solutions, you can't be certain that an Ajax request is going to finish (HTTP makes no guarantees about the success or failure of a request), so if you're relying on the job running. I've seen it kind of work, but it's not the best choice. Forking a process at the end of your request will be a performance nightmare.

Answer 2

Well it depends on the job you want to run. One more complex but more scalable and controllable approach is using Gearman to start and controljobs. The nice thing is you can distribute the jobs too other boxes soyour web server won't get the full load.

The easy approach is using exec and nohup like in

<?php
exec("nohup /usr/bin/php script.php >/dev/null 2>/dev/null &");
?>

The important part there is to dettach the output channels from the PHP process. Forreading the result of the process you might store it in a database.

Answer 3

I haven't tested this but here's an idea.

Using curl, asynchronously send a request to newprocess.php (or wherever your new process script is located.) You can look at the function curl_post_async at http://stackoverflow.com/questions/962915/how-do-i-make-an-asynchronous-get-request-in-php on how to make such a request.

You might want to do some authentication to make sure that only you can fire off newprocess.php ...