Limit the number of cron jobs runnging a PHP script

Question

I have a daily cron job that grabs 10 users at a time and does a bunch of stuff with them. There are 1000's of users that are part of that process every day. I don't want the cron jobs to overlap as they are heavy on server load. They call stuff from various APIs so I have timeouts and slow data fetching to contend with.

I've tried using a flag to mark a cron job as running, but it's not reliable enough as some PHP scripts may time out or fail in various ways.

Is there a good way to stop a cron job from calling the PHP script multiple times, or controlling the number of times it is called, so there are only 3 instances for example?

I'd prefer a solution in the PHP if possible. Any ideas?

At the moment I'm storing the flag as a value in a database, is using a lock type file any better as in here http://stackoverflow.com/questions/851872/does-a-cron-job-kill-last-cron-execution ?

Answer 1

I used a dirty way of tracking the number of the scripts being executed via a table in a database. A launched script inserts a row with an id and the field started_time. On exit it removes his row. Too old rows are considered as "failed\dead scripts".

The worker scripts aren't launched by cron directly. Cron launches "the launcher script" every 1 second or so, which checks the table for the number of active workers and spawns more workers if needed.

Such a scheme is working online for me for 2.5 years already. Just grabbing some constantly updated content.

Answer 2

I had a similar problem and ended up with writing my own little daemon as it seemed to be the easiest solution.

Answer 3

If you have command line access to run scripts from the shell, I'd be inclined to write a script something like I describe in http://stackoverflow.com/questions/1006891/how-feasible-is-a-daemon-written-in-php-using-ignore-user-abort-and-set-time-lim/1008736#1008736 - andthe blog post that links to. In that answer, I describe a long-running PHP script that processes data. If/when it exits, the bash script that is is wrapped in, restarts it so it can complete its work.

When the PHP script has decided that it has done all of it's work for the day, it calculates how many seconds till it needs to run again and then just calls sleep($howLongToWaitTillRunAgain);