I have a databases table with ~50K rows in it, each row represents a job that need to be done. I have a program that extracts a job from the DB, does the job and puts the result back in the db. (this system is running right now) Now I want to allow more than one processing task to do jobs but be sure that no task is done twice (as a per...
It's probably the tenth time I'm implementing something like this, and I've never been 100% happy about solutions I came up with. The reason using mysql table instead of a "proper" messaging system is attractive is primarily because most application already use some relational database for other stuff (which tends to be mysql for most o...
Does anyone have good links on how/when/why to use job queuing to scale web apps? Also, articles on db sharding would be useful too :) ...
I have a Grails app that has some computationally intensive optimizations with a running time of ~5 minutes (perhaps more). Currently, I'm doing these in the main request thread, i.e. it takes 5 minutes for the request to return. It works, but is of course horrible from a usability perspective. So what's the best way to implement this i...
Do you know/use any distributed job queue for python? Can you share links or tools ...
Hi All, I looking into the concept of queueing for web apps (i.e. putting some types of job in a queue for completion by a seperate worker, rather than being completed in the web request cycle). I would like to know if there are any good solutions existing for this which can be utilised in an ASP.NET MVC environemnt. Has anyone had an...
Hi, I'm writing a web application using the Catalyst framework. I'm also using a Job Queue called TheSchwartz. I'm wanting to use a job queue because I'm wanting as much of the application specific code decoupled from the web application interface code. Essentially the whole system consists of three main components: GUI (Catalyst ...
A web application I am developing needs to perform tasks that are too long to be executed during the http request/response cycle. Typically, the user will perform the request, the server will take this request and, among other things, run some scripts to generate data (for example, render images with povray). Of course, these tasks can ...
A web app that launches cpu intensive background jobs in the background whenever users call it. example: youtube.com. people upload videos, it gets converted in background job. how can you deal with many users simultaneously attempting to launch their own background jobs ? it must be queued correct? additionally, if the background job...
Does anyone know of an app that can monitor a beanstalkd queue? I'm looking for something that shows stats on tubes and jobs, and allows you to inspect the details. I'm not really picky about language/platform, just want to know if there's something out there before I write my own. ...
I have a Django application. One of my models looks like this: class MyModel(models.Model): def house_cleaning(self): // cleaning up data of the model instance Every time when I update an instance of MyModel, I'd need to clean up the data N days later. So I'd like to schedule a job to call this_instance.house_cleaning() ...
I originally posted this question over at the Zend Forums but figured it would also be wise to post here. What has to happen, exactly, for a job to retry? I've tried timeouts, 50x response codes, and setting the status to "FAILED". I have a simple script that creates a job... $job_url = 'http://localhost/consumer.php?time=' . microtim...