How have you set-up one or more worker scripts for queue-oriented systems? How do you arrange to startup - and restart if necessary - worker scripts as required? (I'm thinking about such tools as init.d/, Ruby-based 'god', DJB's Daemontools, etc, etc) I'm developing an asynchronous queue/worker system, in this case using PHP & Beansta...
Hey All, I am new to Gearman. I have following queries about optimized Gearman usages: 1) I think worker expects job as a "string". Can I send an Array or Hash as Job to worker? If not, is JSON the best Job format to send Array/Hash? 2) Can I attach rails application routes as worker for Job Server? 3) How do you check if a worker ...
Hello, I did some reading about Gearman and i am wondering if it can be used as a worker pool for background processing. I am interested in a PHP worker pool with the following properties: A. Workers should be able to run concurrently That is, i should be able to have a worker pool and i am expecting gearman to dispatch multiple work...
Riddle me this... in the while($row = mysql_fetch_assoc($result) and $runningOK) loop, if the PHP && operator is used in place of the and the mysql_fetch_assoc fails terrible and returns just the number 1 when running. I've tried mysql_fetch_array() and in place and I still have the 1 problem. It is when, and only when, I replace the && ...
I have a number of Gearman workers running constantly, saving things like records of user page views, etc. Occasionally, I'll update the PHP code that is used by the Gearman workers. In order to get the workers to switch to the new code, I the kill and restart the PHP processes for the workers. What is a better way to do this? Presuma...
At work, we need to build a jobs server for things like sending emails, building PDFs, crunching some data, etc. Obviously, we'd like to build on some sort of generic queueing system. I'm familiar with Gearman, and that this is exact problem that it tries to solve: putting jobs on a queue where workers come to pick them up. However, I'...
Hi, I'm looking for a general overview of best practice to implement a "job allocation" system like Gearman using RabbitMQ messaging system. So basically I'd have something reading the queue and forking? instances to run a job? Any insight is appreciated. ...
Hello everyone. I need to set up a job/message queue with the option to set a delay for the task so that it's not picked up immediately by a free worker, but after a certain time (can vary from task to task). I looked into a couple of linux queue solutions (rabbitmq, gearman, memcacheq), but none of them seem to offer this feature out of...
We are developing a web application wherein with about 50% of write requests, we end up pushing data to multiple data stores and inserting and updating significant number of records in those data stores. To improve the response time, we want to process such requests asynchronously in the background. Our web application is being written ...
One of the characteristics I love most about Google's Task Queue is its simplicity. More specifically, I love that it takes a URL and some parameters and then posts to that URL when the task queue is ready to execute the task. This structure means that the tasks are always executing the most current version of the code. Conversely, my ...
I have 2 machines running same workers. One machine shoud be "primary" as it is very powerful, and the other machine should server as a backup for when primary machine goes down or crashes. When primary machine is up and running, all jobs should default to the primary machine for as long as there are available workers. From my tests, I...
I am attempting to write a python daemon that will launch at boot. The goal of the script is to receive a job from our gearman load balancing server and complete the job. I am using the python-daemon module from pypi (http://pypi.python.org/pypi/python-daemon/). The nature of the job that it is completing is converting images in the or...
Anybody have experience or thoughts on a scheduler to couple with a persistent Gearman queue? Obviously there's crontab available, but i'd like some more functionality. We're looking at Java Quartz. Please weigh in if you've done this before in any capacity! ...
I want to be able to query a gearman server to determine how many instances of a worker I have running (basically I want to make sure that RunTaskA is available and RunTaskB is available if there are no workers handling those tasks, I want to be able to send an alert out. Is there any way to do this? Also: Mad props if you know of a P...
I was wondering if anyone else has figured out how to solve this problem. Whether I install the extension via PECL or compile it by hand, I get the same error: PHP Warning: PHP Startup: Unable to load dynamic library '/usr/local/zend/lib/php_extensions/gearman.so' - dlopen(/usr/local/zend/lib/php_extensions/gearman.so, 9): no suitable...
For now I just want to use Gearman for background processing. For example, I need to email a recipient that they have a private message waiting for them once the sender submits their message into the DB. I assume I can run the worker/client and server on my primary server but I have no problems offloading some of the tasks to a different...
Hi everybody, I have noticed a lot of people discussing Gearman and it's scheduling features making it enable to distribute work onto other servers. However, I have not yet seen a comparison to native cronjobs. What are the differences between cron and Gearman? ...
Hi everybody, I just manage to install gearman on my local Ubuntu test environment. I wonder if I could in any way run both the worker and the client at the same computer, while in development stage? Thanks a lot! ...
I have a number of gearman clients sending a job, say job1. $client = new GearmanClient(); $client->addServer(); $client->doBackground('job1', 'workload'); It takes, say 10 seconds to process this job. I want to track how many 'job1' jobs are waiting for a worker to work on them at any given time. How can I do that? ...
Consider a PHP web application whose purpose is to accept user requests to start generic asynchronous jobs, and then create a worker process/thread to run the job. The jobs are not particularly CPU or memory intensive, but are expected to block on I/O calls fairly often. No more than one or two jobs should be started per second, but due ...