Trying to determine best design for this workflow - c# - 3.0

I might setup a message queuing and processing system for this.

One process/thread/service will monitor the FTP server, and when new files appear, will grab them and dump them into a queue (possibly MSMQ, or just a staging folder, etc.)

Another process monitors this queue, and when a file appears, it grabs it and does watermarking/metadata/etc., then drops it in another queue/folder.

Another process monitors this queue and grabs new files for zipping. After zipping, drop in another queue.

...and so on.

You can setup "work dispatchers" at the end of each queue to grab files and dispatch them to however many worker threads you want.

You don't necessarily have to split it out into this many separate processes and queues - that's up to you to decide. The "queues" can be implemented in a number of different ways as well. You could look at MSMQ as a start, but you might also consider just moving files between folders, etc. WCF and Windows Workflow Foundation might be good technologies to look at first.

Andy, If my volume is large can i get performance of processing may files at once for a particular stage. for instance can i move say 100 files to a Q that is for watermarking and such that the listener of that Q is working on as many files it can at any one point?

ltech 2010-04-01 21:31:42

I would probably create a thread pool of "worker threads" and another thread to act as a listener/dispatcher. The dispatcher thread would retrieve a single file, then hand it off to a worker thread to process. When the worker is done, it puts the file in the next queue, then goes back into a waiting state in the thread pool to wait for the next item. The dispatcher's job is to monitor the file "queue/folder", and hand of single files to worker threads. WCF can actually do all this for you fairly easily.

Andy White 2010-04-01 21:54:28

Andy, can you elaborate on the WCF or point to a reference?

ltech 2010-04-02 00:00:34

would it be wise to use delegates and invoke them using command pattern?main thread moves the file from ftp to a processing folder and invokes a delegate asynchronously.

ltech 2010-04-02 00:30:38

That would definitely be an option. I would probably look at a higher level though, at something like WCF. WCF will provide everything you need to handle MSMQ and threading, without you having to do anything extra.

Andy White 2010-04-02 01:36:28

ansaurus

tags:

views:

answers:

Trying to determine best design for this workflow - c# - 3.0

related questions