processing large number of files - any library or utility to help?

Question

I'd like to process a large number of files - think video transcoding, like youtube, where you have to transform everything into FLV format or image processing where you create thumbs from large RAW files.

Is there any service or library that can help on such processing? I don't mean actual transcoding, but organizing, launching the tasks, monitoring them and handling errors somehow.

Ideally on windows but linux could also be fine.

Update: I'd like utility/lib to handle such tasks at higher level. Dispatch multiple processes, handle and report any failures somehow, etc.

So, in case of transcoding, I'd like such lib/utility to get a list of files to transcode and then it would handle starting the trancoding in multiple threads/processes. You would be able to define what to do on successful completition of the task. And you'd also be able to do this in case of a failure.

Answer 1

Command shells (the Command Prompt on Windows, sh/bash/csh etc on Unix) and scripting languages can all trivially do the same operations for all files matching some pattern.

For example, on Windows 2000 or so onwards:

for %f in (*.jpg) do ... (e.g. call imagemagick's convert to generate a thumbnail)

To get the for help, at the command prompt do for /?

If you want to exploit parallelism, e.g. spin off six at a time to utilise all six of your CPU cores, then you can use the unix 'make' program, which is also available on Windows. You have to make a Makefile specifying what you want to do, and execute make with the --jobs switch

An alternative to the make approach for parallelism is to use a scripting language. Many scripting languages easily support running multiple child processes at once e.g. Python's Popen().

Answer 2

If I was going to do this I would create a .NET program to watch a directory for incoming files and then based on their file type fire off something like http://ffmpeg.org/ to convert the video into an output directory or for the images using something like ImageMagick.

Monitoring the processes might require some thought although it should be possible to keep an eye on a shell task that you have created, or at the very least check for a file in the output directory that you are expecting to have been created.