I'm working on a simple web crawler in python and I wan't to make a simple queue class, but I'm not quite sure the best way to start. I want something that holds only unique items to process, so that the crawler will only crawl each page once per script run (simply to avoid infinite looping). Can anyone give me or point me to a simple qu...
Is there a method/pattern/library to do something like that (in pseudo-code): task_queue.push_back(ObjectType object1, method1); task_queue.push_back(OtherObjectType object2, method2); so that I could do the something like: for(int i=0; i<task_queue.size(); i++) { task_queue[i].object -> method(); } so that it would call: obj1...
Hi, I have an array of files in the object FileInfo[]. In my service, I am currently looping through the files and processing them. I want to loop through the files, and remove them from the collection as they are processed. I think a queue is ideal for this, but how can I loop through a queue collection? (never used a queue in C# ...
Hi, I have an array of items, sorted so that the oldest item is first in the array. I want to load a queue from the array, such that when I pop the items on the queue the oldest item comes first. How can I do this? ...
I have a table I'm using as a work queue. Essentially, it consists of a primary key, a piece of data, and a status flag (processed/unprocessed). I have multiple processes trying to grab the next unprocessed row, so I need to make sure that they observe proper lock and update semantics to avoid race condition nastiness. To that end, I'...
Now, I could go ahead and write this using FileSystemWatcher etc, but before I start coding - I wonder if there is a better option that I can re-use (rather than re-invent). From my web-server, I want to drop files (to be printed) into a network share as a queue (ordering not hugely critical). Ideally, a windows service (or other client...
I am designing a system to deal with an external web service. The service limits the number of requests that can be made over a certain period of time (T). The system allows for the batching of a certain number of requests (R). There are a certain number of operations that the service supports (O). My code will process an unknown num...
I've this system that consists of lots of different stages piped together. Specifically, I've got lots of different processes that communicate via tcp-ip; messages are exchanged via a packet oriented protocol (more messages can be stuffed together in a single wire-protocol data unit if certains conditions are met); I've control flow queu...
I want to do a breadth first search of a tree using a Queue var q = new Queue<T>(); q.Enqueue(Root); foreach(T root in q) { foreach(T t in root.Children) q.Enqueue(t); } However I get a "Collection was modified after the enumerator was instantiated." Exception. Is there a C# type that I can do this with? Edit: a little rea...
There's a name for this, but I don't know what it is so it's hard to google. What I'm looking for is something in the java concurrency utilities, which is a pair of queues, an "pending" queue used by a producer, and a "processing" queue used by a consumer, where the consumer can swap the queues atomically. If used in this way (1 produce...
Hi, I am using a application where lower level application always invokes a callback RecData(char *buf) when it receives the data. Now in the call back i am creating two threads and pass the consumer and producer function to these created threads respectively. My code looks like as below: void RecData (char * buf) { CreateThread(NULL,...
Hi, we have a web application that does various things and sometimes emails users depending on a given action. I want to decouple the http request threads from actually sending the email in case there is some trouble with the SMTP server or a backlog. In the past I've used JMS for this and had no problem with it. However at the moment f...
I'm working an image processing website, instead of having lengthy jobs hold up the users browser I want all commands to return fast with a job id and have a background task do the actual work. The id could then be used to check for status and results (ie a url of the processed image). I've found a lot of distributed queue managers for r...
I've got a thread dump of an applet running on JVM 1.6.0_12 in Opera 9.64 Build 10487 - and it shows three event queues! As far as I know the Java Swing event handling is single threaded - did this change in any recent update? My problem is, that multiple event queues will tend to cause deadlocks since I've got some more locks than only...
I have several animations that I want to perform on different objects in the dom. I want them to occur in order. I do not want to do it like this: $('#first').show(800, function () { $('#second').show(800, function () {...etc...}); I want to add all my animations(and cpu intensive functions) to some kind of queue object that will m...
Are there any good examples of conflating queue in dotnet. I have thousands of messages per second coming in from another system and i wanted to see if this was the best solution and see some implementation examples ...
I have a robot that is contolled with a local UI via a 9 pin serial connection and I would like to make it controllable via a web page, but only one user should be able to interact with it at any time. I'm still thinking about how to use WCF communications between the web server and the local PC, and might ask about that at a later time...
Consider a system where jobs are queued and processed in FIFO order. They currently wait an average of N seconds before being processed. If jobs take an average of M seconds to process, what will be the impact to N if we reduce the processing time to M/2 seconds? ...
I created a RESTful service using WCF which calculates some value and then returns a response to the client. I am expecting a lot of traffic so I am not sure whether I need to manually implement queues or it is not neccessary in order to process all client requests. Actually I am receiving measurements from clients which have to be sto...
Hello, I use MSMQ through WCF to store messages in private queue. (private queue hosted by Windows Server 2003). Messages are stored only for 24 hours and automatically removed from queue, somebody know why ? Thank's for your help ...