views:

223

answers:

1

I have a simple messaging queue setup and running using the Zend_Queue object heirarchy. I'm using a Zend_Queue_Adapter_Db back-end. I'm interested in using this as a job queue, to schedule things for processing at a later time. They're jobs that don't need to happen immediately, but should happen sooner rather than later.

Is there a best-practices/standard way to setup your infrastructure to run jobs? I understand the code for receiving a message from the queue, but what's not so clear to me is how run the program that does that receiving. A cron that receives n messages on the command-line, run once a minute? A cron that fires off multiple web requests, each web request running the receiver script? Something else?

Tangential bonus question. If I'm running other queries with Zend_Db, will the message queue queries be considered part of that transaction?

+1  A: 

You can do it like a thread pool. Create a command line php script to handle the receiving. It should be started by a shell script that automatically restarts the process if it dies. The shell script should not start the process if it is already running (use a $pid.running file or similar). Have cron run several of these every 1-10 minutes. That should handle the receiving nicely.

I wouldn't have the cron fire a web request unless your cron is on another server for some strange reason.

Another way to use this would be to have some backround process creating data, and a web user(s) consume it as they naturally browse the site. A report generator might work this way. Company-wide reports are available to all users but you don't want them all generating this db/time intensive report. So you create a queue and process one at a time possible removing duplicates. All users can view the report(s) when ready.

According to the docs it doens't look like the zend db is even using the same connection as your other zend_db queries. But of course the best way to find out is to make a simple test.

EDIT The multiple lines in the cron are for concurrency. each line represents a worker for the pool. I was not clear, you don't want the pid as the identifier, you want to pass that as a parameter.

          • /home/byron/run_queue.sh Process1
          • /home/byron/run_queue.sh Process2
          • /home/byron/run_queue.sh Process3

The bash script would check for the $process.running file if it finds it exit.

otherwise:

  • Create the $process.running file.
  • start the php process. Block/wait until finished.
  • Delete the $process.running file.

This allows for the php script to die but not cause the pool to loose a worker.

If the queue is empty the php script exits immediately and is started again by the nex invocation of cron.

Byron Whitlock
Follow up questions; (apologies if this is naive/obvious). Are you suggesting I1. Write a PHP program to receive the messages;2. Write a shell script that looks for a $pid.running file created by, the PHP script?;3. If the $pid.running file isn't found kick off the PHP script and then, exit?;4. Have many lines in the crontab for kicking off the shell script. Some of these will run, see the $pid.running file and exit, but by having multiple attempts running each minute, you ensure a constant running/re-running of the program at a regular basis;Do I have it, or am I missing something?
Alan Storm
almost. see my edit.
Byron Whitlock
I also forgot, you can also set a timeout on the running file by looking at its creation date.. You would need to have your php scripts process only x number of entries and exit so you would know how long the longest running script would take.
Byron Whitlock
Ah ha, I thought I was missing something. Thanks for taking the time to explain that in newbie levels of detail for me!
Alan Storm