MSMQ with dynamic priorities

Question

Hi.

I'm doing a project with some timing constraints right now. Setup is: A web service accepts (tiny) xml files and I have to process these, fast.

First and most naive idea was to handle this processing in the request dispatcher itself, but that didn't scale and was doomed from the start.

So now I'm looking at a varying load of incoming requests that each produce ~ 50 jobs on my side. Technologies available for use are limited due to the customers' rules. If it's not Sql Server or MS MQ it probably won't fly.

I thought about going down the MS MQ route (Web service just submitting messages, multiple consumer processes lateron) and small proof of concept modules worked like a charm.

There's one problem though: The priority of these jobs might change a lot, in the queue. The system is fairly time critical, so if we - for whatever reasons - cannot process incoming jobs in a timely fashion, we need to prefer the latest ones.

Basically the usecase changes from reliable messaging in general to LIFO under (too) heavy load. Old entries still have to be processed, but just lost all of their priority.

Is there any manageable way to build something like this in MS MQ?

---

Expanding the business side, as requested:

The processing of the incoming job is bound to some tracks, where physical goods are moved around. If I cannot process the messages in time, the things are "gone".

I still want the results for statistical purpose, but really need to focus on the newer messages now.

Think of me being able to influence mechanical things and reroute things moving on a track - if they didn't move past point X yet..

Answer 1

So, if i understand this, you want to be able to switch between sorting the queue by priority OR by arrival time, depending on the situation. MSMQ can only sort the queue by priority AND by arrival time.

Although I understand what you are trying to do, I don't quite see the business justification for it. Can you expand on this?

I would propose using a service to move messages from the incoming queue to a number of work queues for processing. Under normal load, there would be a several queues, each with a monitoring thread. Under heavy load, new traffic would all go to just one "panic" queue under the load dropped. The threads on the other work queues could be paused if necessary.

Cheers
John Breakwell