I'm using the Python GAE SDK. I have some processing that needs to be done on 6000+ instances of MyKind. It is too slow to be done in a single request, so I'm using the task queue. If I make a single task process only one entity, then it should take only a few seconds. The documentation says that only 100 tasks can be added in a "batc...
I have printer in my network to which I get by ip. I use OLEPRNLib library to get it status and it works ok. I try to get job queue with this but it won't works: System.Management.ObjectQuery oq = new System.Management.ObjectQuery("SELECT * FROM Win32_PrintJob"); ManagementObjectSearcher query1 = new ManagementObjectSearch...
Hello everyone, I would like to build a priority queue in python in which the queue contains different dictionaries with their priority numbers. So when a "get function" is called, the dictionary with the highest priority(lowest number) will be pulled out of the queue and when "add function" is called, the new dictionary will be added ...
Hi, here is what I'm trying to do: I set up a MapReduce job with the new Mapper API. This basically works fine. The problem is that the Task Queue retries all tasks that have failed. But actually I don't want him to do that. Is there a way to delete a task from the queue or tell it that the task was completed successfully? Perhaps pass...
Google documentation on Bucket_size parameter gives just a brief definition, a Wikipedia link and an example: bucket_size Limits the burstiness of the queue's processing, i.e. a higher bucket size allows bigger spikes in the queue's execution rate. For example, consider a queue with a rate of 5/s and a bucket size of 10....
I'm working on a GAE app that will take a comma separated list and create a batch job out of it. I want to add one task to the task queue from any one batch job at a given time. The problem with this is that if a task can't be executed it will just keep repeating for ever and the rest of the tasks will never get added to the queue. Is th...
Short question: I would like to spawn a single background thread that would process work items submitted to a queue (like a threadpool with one thread). Some of the work items are capable of reporting progress, some are not. Which one of the .NET's myriad of multithreading approaches should I use? Long explanation (to avoid asking ab...
I am looking for a WorkQueue API offering the following features: java.util.Queue compatible offers (optional) Set-semantic single- and batch-processing concurrency (of course) scheduling different processing policies wait until next scheduled execution pre-process, if batch size met delayed processing (minimum time in queue, before b...
Consider a PHP web application whose purpose is to accept user requests to start generic asynchronous jobs, and then create a worker process/thread to run the job. The jobs are not particularly CPU or memory intensive, but are expected to block on I/O calls fairly often. No more than one or two jobs should be started per second, but due ...
I have a basic queue of tasks executing (c# WinForms App that talks to 3 separate systems). Everything is great until one of the web services decides not to respond in with the usual speed. I'm not interested in speeding things up via multi threading the jobs, but now that I am using it in production, I can see the benefit of having at...
From Task Queue Python API Overview: If your task's execution nears the 30 second limit, App Engine will raise an exception which you may catch and then quickly save your work or log process. Which exception is that? ...
Hi. I'm planning on using GAE's TaskQueue API to refresh my cache of HTML pages, which I save in the Datastore (and memcache, but Datastore is more reliable). Once a week, i add/edit some data, and need to regenerate associated HTML pages, and triggering tasks is the way to do. Note that different edits may imply changing the same c...
Google app engine task queues have configuration as (example) <queue> <name>mail-queue</name> <rate>5/m</rate> <bucket-size>10</bucket-size> </queue> Here, what does the 'bucket-size' mean? I could not find a comprehensive documentation about this in google app engine documentation. Does specifying this as 10 means th...
I just watched Batch data processing with App Engine session of Google I/O 2010, read some parts of MapReduce article from Google Research and now I am thinking to use MapReduce on Google App Engine to implement a recommender system in Python. I prefer using appengine-mapreduce instead of Task Queue API because the former offers easy it...
What correlation is there between datastore and task queue downtime? (I'd like to use the task queue to defer some operations in the case of datastore downtime.) ...
I am using celery for distributed task processing. I wanted to deploy my work on a web-host, just to show the working of my project. So how can i get djcelery to use a database (sqlalchemy) as backend with django instead of rabbitmq or other ampq servers. ...
I am using a task (queueing-task) to queue multiple others tasks — fanout. When I try to use Queue.add with task argument being a list of Task instances with more than 5 element's and in transaction… I get this error. JointException: taskqueue.DatastoreError caused by: <class 'google.appengine.api.datastore_errors.BadRequestError'...
I have task queue that adds task to second queue, while second queue is paused. Is it possible to add task to task-queue which is paused? ...
I know you can view the currently queued and running tasks in the Dashboard or development server console. However, is there any way to get that list programmatically? The docs only describe how to add tasks to the queue, but not how to list and/or cancel them. In python please. ...
Hi everyone, I'm using GAE and task queue. In queue.yaml file, I keep default setting: 5/s. 1 month ago I thought it's enough but now there are about 40-50 tasks in one queue so my application runs too slow. I want to know how many tasks per second is enough ? Can I change to 100/s ? Thank you :) Update: My application gets data ...