Windows Services in a web farm (how to prevent concurrency conflicts)

Question

Looking ahead, to when a Web Farm will be created...

We have various windows service applications performing various back-ground tasks: sending emails, thumbnailing/processing files etc.

In a server farm environment, we could in theory install and run each service on each machine, however there may be conflicts due to business object based "first-write-win" or undesirable duplication of tasks eg: the same emails being send a number of times.

The "first-write-wins" scenario can be overcome through code or logic that resolves the issue or skips the item. (if its a basic queuing mechanism (query list and process record) it will just be tried again)

For the duplication of tasks, one could perform tasks in transactions to prevent other processses grabbing the same resources, however I guess they will just wait and we would need to handle some login based on things happening again? Would this be the case?

To prevent duplication of tasks I am thinking of maybe building some mechanism that writes a "locking flag" to the database that will be set by the first service that fires up. This flag will be dated and reference the instance of the service. The service that "owns" the flag will run the tasks and periodically update this date. There will also be a timeout/tolerance setting on each service in the situation where the service doesnt own the flag, it will check the tolerance and take over if required.

Does this sound like a resonable solution or should one go to a better message queuing based solution at that stage (also not sure how that would work in a farm environement?

Tecnologies: C# / ASP.NET / SQL Server

Answer 1

I can think of two mechanisms.

1. If all your tasks are derived from database activites you could create a new "tasks" table and set triggers on the activities so that a new record with the task information is inserted. Your windows services can then poll the "tasks" table and use an atomic select and delete transaction to ensure only one service does it.

2. You could use a message queue and drop your tasks on it and then have your windows services poll the queue. Each queued message will only be able to be dequeued by a single service so you can ensure each task is only done once. Of course you then have to worry about load on your message queue. Depending on your application you could use something like Amazon's Simple Queue Service which would alleviate you having to worry about reliability and scalability.