Throttling MSMQ messages / prioritising messages

Question

I'm not sure how best to describe this, or the best title, so bear with me!

I'm using MSMQ to queue up a large number of commands that will fire off HTTP requests to various websites and APIs. In order to avoid hammering these services (and to stay within certain pre-defined request limits) I need to ensure tasks hitting the same domain are executed only after a minimum time has elapsed from the last request.

Previously I was using a database to queue the tasks, and so could perform a query to achieve this, but we rapidly outgrew that solution with the number of tasks (way too many deadlocks on the table).

Does anyone have any suggestions as to what approach we could take? I've considered just keeping taking items off the queue until you find one that you can execute - but I'm conscious there's nothing to stop the ordering on the queue meaning we could take off thousands for the same domain before finding one on a different domain.

Thanks!

Answer 1

You 'outgrew' relational database and replace it with MSMQ? This is like 'promoting' from college to kindergarten! 4GB size limitation, no high availability solution, questionable recoverability, no querying of the message queue, you are already running into the very limitations than everyone is claiming when moving away from MSMQ to a relational database based queuing...

I highly recommend you go back to the table based queuing. There are ways to avoid deadlocks in table based queuing, specially with the advent of OUTPUT clauses for DELETE (which is the key to write a fast deadlock free dequeue operation from a table based queue).

Or you can use an off-the-shelf in database queuing solution like Service Broker. Actually SSB would offer a free lunch with many of its features, like built-in throttling (MAX_QUEUE_READERS), enable/disable processing on a schedule (by enabling/disabling activation), built-in timers for retry, high-availability based on database mirroring or on clustering, or built-in priority handling.