Processing long-running operations from a windows service.

Question

Edit (again): Let me simplify my problem. I have a Windows Service that exposes some WCF endpoints with methods like:

int ExecuteQuery(string query) {
   // asynchronously execute query that may take 1 second to 20 minutes
   return queryId;
}

string GetStatus(int queryId) {
   // return the status of the query (# of results so far, etc)
}

What is the best way to implement the ExecuteQuery method? Should I just call ThreadPool.QueueUserWorkItem to get my query going?

Note that the actual work behind executing a query is done by load-balanced black box. I want to be able to have several queries going at the same time.

The analogy is a web browser that is downloading multiple files simultaneously and you have a download manager that can track the status of each file.

Answer 1

Take a look at Microsoft Message Queuing (MSMQ):

Microsoft Message Queuing (MSMQ) technology enables applications running at different times to communicate across heterogeneous networks and systems that may be temporarily offline. MSMQ provides guaranteed message delivery, efficient routing, security, and priority-based messaging. It can be used to implement solutions for both asynchronous and synchronous messaging scenarios.

It's good to know that Windows Communication Foundation (WCF) can leverage queuing services offered by MSMQ.

Answer 2

Adding to the MSMQ answer, you could think about looking at using an Enterprise Service Bus (ESB) to handle these sorts of things, if future scalability is a concern. Check out NServiceBus for one .NET example.

Answer 3

I use a Windows service for a very similar task and it works very well. I use database tables to queue requests and responses, as it gives me a persistent queue that can be accessed over the network from remote ASP.Net applications, and concurrency control through transactions.

A supervisor thread on a timer spawns workers whenever incoming requests need servicing. I use a separate database tables for configuration and control so that I can administer the service and pause the supervisor from an application without while leaving the service core running. Logging to a separate table is a convenient way to see what's happening from web apps and a local admin app.

I wouldn't use the ThreadPool for long-running threads, but instead create a worker class that runs in its own thread and uses callback methods to update the supervisor with progress and completion status.

Answer 4

I would use WWF (4.0):

You can start long running transactions that can be handle in a few machines, execute task in parallel, failure support, friendly coding, you can manage it with appfabric, it is free...

Answer 5

Either this is a trick question or a no-brainer... ThreadPool.QueueUserWorkItem is about the easiest way to go when you want to execute a piece of code concurrently. I'm sure you already knew that, so technically you have already answered your own question.

So if this is not a trick question, then are you asking exactly how to pass the query in the ThreadPool.QueueUserWorkItem?