C# - Queue and multithreading

Question

I am very new to multi-threaded programming. Following is what I am trying to achieve:

• Create a windows service that continuously reads the database (or somekind of message queue, please suggest what would be best) for a entry in a table.
• Look for a new entry in the table
• If there is a new entry, check if there is enough thread in the threadpool (not sure how that would work), start a new thread and do some work. There could be many new entries during high traffic thats why I need to have it multi-threaded.

Thank you all. Please help me with ideas and link. I appreciate your help.

Answer 1

If possible, you might want to consider having the database send signals on modification. Continuous reads seems like a backwards design. If you don't want to create your own service, you might want to use Windows Scheduling Services to launch your application to read the database. You wouldn't be able to use a ThreadPool to limit the number of concurrent actions though. But if you do create your own service, using the ThreadPool class is pretty straight forward. You could simply set a thread limit and keep queuing work items based on database updates. Also play around with the SetMinThreads, to keep the threads active and reusable.

Answer 2

If you are using SQL Server you might want to use SqlCacheDependency Class to monitor the database changes.

Answer 3

All you'll need is a ThreadPool and MSMQ (forgive the link to a VB.NET Article); very trivial. What you've written up in your example is polling, not a queue, technically.

Answer 4

Where is this table getting updated from, and why?

If you are just using this table as a way to 'communicate' between 2 applications, then I would suggest using an EMS system like MSMQ. You can have your 2 applications talk to eachother and have all the guareteed delivery goodness that comes with MSMQ. It's also baked into the WCF and fairly easy to integrate.

If that table is being updated for some other purpose (e.g. you are not trying to build your own queueing system) then I agree with Rashmi on the SqlCacheDepdency recommendation. This will essentially allow you to hook your .NET code into SQL so that an event is raised when a new record in the table is added or an existing one updated (or anything changes the result set for that matter).

Answer 5

As others have said, you'd probably be best served using MSMQ. I'd take it step further and encourage you to write your service using WCF and deploy it to IIS. It's not exactly easy, but Microsoft has already done so much of the work that you'd like to do.

Tom Hollander has a great series of articles on setting up MSMQ, WCF and IIS here:

http://blogs.msdn.com/tomholl/archive/2008/07/12/msmq-wcf-and-iis-getting-them-to-play-nice-part-1.aspx

Answer 6

I've coded precisely this kind of Windows service, and I think you're on the right track. I use a database table for my queue, which gives me a persistent queue in case of a system crash, readily allows multiple writers from across the local network, and lets me manage contentions through transactions.

I create separate threads for each process and keep track of the number active threads, which lets me adjust on the fly. Since each thread makes a web service call, there was negligible benefit from using the threadpool, and this also allowed me to start a lot of threads since they blocked waiting on an external server.

It's easy to manage if you allow only one reader, thus allowing you to keep track of the queue entries you've read by just using an identity column and selecting the next one higher than the last one. As a thread finishes it can delete the entry it was working on. If your service crashes it will just pick up with the first unread entry. You can get fancier by including a column that indicates when the entry was touched, and if the processing fails (perhaps due to a network timeout) then you can untouch it and let it get caught by the next read. If you need to have multiple readers then use a timestamp field that indicates when it was touched; one reader can recover for another crashed reader by checking when an entry was touched.

You should also trap the service-closing event so that you can finish processing cleanly, and include your own logging so that you can trace what's happening once you get several simultaneous writers.

Also consider a test version that lets you hammer your queue reader and measure how it bears up under load. This will go a long ways to helping you weed out the not-so-obvious gotchas that lurk in a multi-threaded application.

But I had a blast creating this and I hope you do, too.