Service to process queued long-running processes - where to start?

Question

What's the best way to build a service to handle queued long-running processes? For example, this is what we're trying to do

1. User uploads 401k data to web
2. Data goes into processing queue (a database table)
3. 401k is processed (could take a couple minutes per client)
4. User is informed via e-mail that 401k was accepted

We could certainly write a service to handle this, but what if the service fails? How would someone be notified? What if there is an exception? Also, should this service handle other non-user-initiated processes (like sending e-mails and such).

We have a team of skilled .NET developers, but all of our experience is in web/client apps or console apps launched by the task scheduler, so any direction you can provide would be much appreciated.

Answer 1

Consider using MSMQ (System.Messaging) for this task. Perhaps a flow like this:

• User uploads data to a website.
• data is written to an MSMQ.
• at interval n, a Windows Service peeks/reads from the queue for the next waiting message.
• the work is done by the service, data is written to a DB, and it sends another message to another queue for notifying the customer.
• another service peeks/reads from the 2nd queue. Sends email/notification to the customer as required. Suggest that this 2nd queue is needed to handle SMTP outages, etc., and can be managed independently from the 1st queue.

Your other concern around potential failings could be yet another service, or perhaps a polling mechanism in your website. Read the DB for the datetime of the last processed message. Read the number of items 'waiting'. If the numbers aren't satisfactory, send email to admins/customers/etc. as necessary.

Using MSMQ means you get to use a pull-system, and won't be having to burden your database & network by polling every n against the database. The message send are all transactional, so you won't have to worry at all about lost/unAck'd mssages.

@Jess: Understood you're trying to cover bases. A proper queue would help in that it's not hammering your database with requests. The scale of your app/project/customers will dictate on whether this is an issue. Indeed, you could have thrown this all into a single .aspx in Page_Load, but you definitely know better.

Re: overkill. I 'read' into the question that there were some concerns about downtimes and how such an app could handle this. A web service, out of the box, doesn't:

• handle DB outages well, as it's reliant on the DB to save the output of its work. It's the 'catcher', the 'worker', and the 'resulter' all in one shot. Does this really belong in the DMZ? Note Item 1: "Web", not intranet.

• scale without adding more web+worker nodes.

This suggested solution involves:

• MSMQ has its own storage implementation, so it's not reliant on your app database for queueing and order-of-delivery. It's part of Windows, and runs as a service.If MSMQ was down, then Windows is down, or security is not configured correctly.

• asynchronous processing. Your web tier can simply 'catch' the request, and 'put' to the queue. That's it. No spinning up threads, no reliance on application databases. It can get back to the job of receiving requests from other users. Users won't have to 'feel' the lag of a busy workday.

• scalability. You can deploy your Windows Service to 1+ machines to work.

• separation of work: 401k processing, emailing, request-handling + authentication all in distinct modules.

• security - it's not 100% clear whether this in an intranet or internet solution. If internet-exposed, then consider how you want to send messages from DMZ to your internal application. Can you justify having read-write access to your application DB from the open internet? Consider a facade of some kind. The queue provides this.

Answer 2

HI My thoughts

By reading your question , i am sure you need a wrokflow here to support this kind of activity.

Use Windows Workflow since it supports

1- Notifications

2- Saving states

3- Long running processes

i think all this is mapped to your requirement

Answer 3

If your experience is in web apps, do it in the web app. You can have your processing done within the context of the web service instead of in its own.

edit

Apparently I wasn't clear enough. IIS serves up ASP.NET and static content, but it's also an ideal place to host a polling thread that processes the jobs in the database. I call it ideal because it can update static variables that can be displayed in an internal web page, and it can benefit from all of the logging, reseting and other infrastructure that IIS provides. This avoids all of the complexity of creating a separate service just to run one thread, and then have to deal with cost of that solution.

I've done this a few times, with good results.