Suggestions for a pluggable task framework in Django

Question

I am developing a website which is aimed at being a GUI for several image processing algorithms (referred to as 'tasks').

At the moment, only one of these algorithms is finished, but there are more to come (which will have a similar, but not quite the same, workflow)

Basically, the algorithm works as follows (not that it matters a lot, but just for the sake of clarity): 1) Enter some parameters and an input image 2) run algorithm part one --algorithm runs-- 3) review an interim result 4) edit the parameters if needed, and go to 2. 5) run algorithm part two --algorithm runs-- 6) review the result 7) adjust some of the interim results if needed, and go to 5 8) all done

I expect some of the additional tasks to have a rather similar work flow, though it is perfectly possible that they won't.

So, the easy way to implement this would be to create a separate django application for each algorithm.

However, I'd like to be able to browse through a sorted list (by time of completion) of completed tasks, and for each of these task display a summary (name, description, start time, thumbnail).

Is there anyone that can provide me with some suggestions on how I can implement this? I'd like to keep it as simple as possible, so additional task can be 'plugged' in as easy as possible.

Answer 1

I would make an app with a very abstract definition of a Task model. The Task model might contain properties for:

• the input arguments,
• the function to run,
• the time that the task was submitted,
• the time that the task has been actually running, and
• the result (which would be something like a singleton Task.NotFinished until finished).

You could consider using twisted to run the tasks because:

• twisted has a well designed and tested implementation of asynchronous tasks; and
• you could run the tasks on other processors or machines.