Is this a good architecture / design concept for handling/manipulating file uploads?

Question

Hi folks,

I'm looking at adding multi-file uploading to our ASP.NET MVC web application. I am going to use the 3rd Party Multi-File uploader Aurigma to handle the actual uploading.

Once each file is 100% received by the web server, it needs to be checked for the following

1. is it an image or video.
2. if it's an image, does it need to be resized? if so, resize.
3. if it's a video, we need to re-encode it to flash (unfortunately.. bring on html5 :) )
4. finally, store on S3 amazon.

So i know how to do most of steps 1 to 4 - that's not the question.

My question is: how should I really be handling this workflow? Especially if i get a bulk of media at once -> after all, this is a multi-file uploader :) I don't want to have lots of media getting processed at once (eg. 5 video's getting encoded to flash..).

So I thought i might have MSMQ installed and when each file gets 100% recieved, then pop it into MSMQ. That way, only one item is getting processed at once. The cost of this, is that if there's a lot of expensive processing, the queue might actually start to get a few items in there ... and there will be some waiting period from the time an item is uploaded to the site, until it's actually 100% processed. We can easily handle this in the UI to tell them it's not finished yet.

So - does this sound like an OK way to handle this problem?

Secondly, this is in a web farm scenario .. so installation / deployment / maintenance needs to be considered.

Would love to see what other people are thinking.

Answer 1

What about running a windows service that watches temp directory (the directory that the files are uploaded). Then have it spawn X threads of simultaneous processing. That way if you set X to 10, then it will only process 10 files at a time. Once processed, they can be moved to the permanent directory and the service will pick up the next file in the temp directory.

It keeps the processing out of your web application too.

Answer 2

I like your solution.

The MVC layer should be responsible for validating the file and queueing the request only. A background process should do the encoding and other heavy lifting.

With this architecture you gain the ability to have multiple encoding servers if that's where the bottleneck ends up.

I've used MSMQ in the past in similar situations. It's a great "fire and forget" solution, particularly across server boundaries. Obviously you could use web services, shared file system, etc. for the UI -> background processing communication, but I think MSMQ is the right answer.