PHP thread pool?

views:

242

answers:

+4 Q:

I have scheduled a CRON job to run every 4 hours which needs to gather user accounts information. Now I want to speed things up and to split the work between several processes and to use one process to update the MySQL DB with the retrieved data from other processes.

In JAVA I know that there is a thread pool which I can dedicate some threads to accomplish some work.

how do I do it in PHP?

Any advice is welcome.

Thank

+1 A:

PHP is probably not the most suitable language for multi-threading.

You might want to have a look to different solutions. For example, Thrift allows you to have a PHP front-end talking with a Java back-end, where you could easily implement your desired behaviour.

If you still want to do this in PHP, you might want to have a look to:

http://uk2.php.net/pcntl

http://www.electrictoolbox.com/article/php/process-forking/

Roberto Aloi 2010-04-07 11:55:15

Chech these posts - * http://www.alternateinterior.com/2007/05/multi-threading-strategies-in-php.html * http://www.electrictoolbox.com/article/php/process-forking/

Basically you need to share data between processes and as I see, you will probably need to write to some file first. Fetch using the main process (make it a ajax-polling type process) and write to DB.

pinaki 2010-04-07 11:55:31

what do you mean by ajax-polling type process?I think i will take your path and fork some processes to do the taskand append the result to file.The main process will fetch all records available at the moment in the file, clears the file andthen releases the lock and will process them.I will use a lock on the dedicated file to prevent mulpile access to the file.What do you think about this behavior?

embedded 2010-04-07 12:09:54

That behavior's exactly what comes to my mind. in your case, since it is cron-job, it will keep running at specific intervals and gather data from the file and then write to DB. Ajax-polling will be for a webserver approach where you use javascript to make ajax requests instead of the cron job.

pinaki 2010-04-07 12:21:47

I think I'll implement this and use it when I see my cron job execution takes more than 4 hours.In terms of web hosting I need to make sure that the pcntl library is enabled?

embedded 2010-04-07 12:28:14

What's the best way to split the work between 3 processes?I need to go over all records in my DB and do some taskright now I only see the LIMIT option where each child process will query the DB using the LIMIT option and then he will be working on the query results.the main process will wake up every minute checks the file and process the DB with the current file's record.

embedded 2010-04-07 13:18:40

You can fork new processes in PHP too: pcntl_fork()

BTW. is that script running longer than 4 hours? Otherwise I see no reason why complicate it with thread or process management.

Messa 2010-04-07 11:57:31

This can be dynamically changed.it depends on the number of users using my service.I'll use this mechanism ONLY when the script execution will be more than 4 hours

embedded 2010-04-07 12:13:31

PHP and Threads (these 2 words) cannot go together in the same sentence. PHP does not offer thread support. You can try the pcntl forking mechanisms or asynchronous processing which in your case is not helpfull.

You can use a workload distribution mechanism that might be what you want by having a look at Gearman (suggest you google it).

As described by others "it is a distributed forking machine" that can offer the workload distribution that you are looking for in order "to speed things up".

regards,

andreas 2010-04-07 13:26:27

I'll take a look in it.Do web hosting providers support this kit?

embedded 2010-04-07 13:34:52

It's just a lib that you install - server and client - you start the deamon and you are ready to accept requests for processing (highly recommended for distributing workload) - http://gearman.org/ -check the presentations also

andreas 2010-04-07 13:42:46

I think this is way over my needs.I will implement my own mechanism.Thanks

embedded 2010-04-07 13:43:59

As others have said, forking processes is easier than spawning threads with PHP. But why do you think that having a single dedicated thread to write the results back to the database is a good idea? Although this is slightly simpler to do with threads rather than processes, its still a complex overhead which doesn't seem to add any value to the overall objective.

Indeed, its a lot simpler to start up several instances of the script (with some parameter to partition the data) from cron rather than initiating a fork from within the PHP code - and not bother with any bottleneck for recording the data back into the database.

symcbean 2010-04-07 15:32:29

That's a good option which IMO is much better than forking processes.Do you have any idea how to partition the data and how many instances should I run not to overhead the system?I'm thinking on 3 instances.

embedded 2010-04-08 06:39:54

As many instances as you like. If the data is identified by a sequence number or some other numeric then use MOD to get every Nth case, e.g. 'SELECT * FROM mytable WHERE MOD(id, $number_of_instances)=$this_instance_id' (remember to run a 0th instance)

symcbean 2010-04-08 11:49:28

thanks I'll take a look on the MOD option.

embedded 2010-04-08 14:12:18

ansaurus

tags:

views:

answers:

PHP thread pool?

related questions