Does this web app require a task queue?

Question

Background

I have a web app that will create an image from user input. The image creation could take up to a couple seconds.

Problem

If I let the server thread, that is handling the request/response also generate the image, that is going to tie up a thread for a couple seconds, and possibly bog down my server, affect performance, kill puppies, etc.

Question

Should I use a task queue, such as Celery, so that the server can hand off the image creation, and go back to handling requests/responses? I have no problem letting the user who is creating the image wait, but I dont want it to effect other peoples access to the site.

Answer 1

I have an image-generating site as well (Names4Frames) and did things like this via AJAX (and PHP). I haven't had any noticeable slow-downs (or dead puppies), but the site in question doesn't generate huge amounts of traffic either. I am not an expert on threads, and to be honest, I'm not 100% sure what your exact concern is and what technologies you're using...

Basically have one page request the image from another page (Perhaps even located on a different server), and when it's done the second page passes back to the first any relevant information about the image for processing/display purposes. If we're only talking about a few seconds, I can't see that being a real problem, unless you're dealing with MASSIVE amounts of visitors constantly using this image creation service.

Answer 2

I'm going to say No - for now.

• A couple of second is not that long.
• You'll anyway have to implement some sort of polling (or comet processing) to feed the image back to the user.
• It will make your system more complex.
• Design the system so adding on a task queue later on is feasible and easy.

So, keep it simple at first and get it working, but Keep in mind that you might add a task queue later.

Implement that task queue when/if you need to scale.

Answer 3

rule of thumb: use a queue if tasks could pile up.

In your case, the task could take up to 2 seconds, assuming 8hours a day, you could do up to 8*60*60/2 = 14400 images a day without concurrency. If you get over 7200 requests a day, you have a 50% chance of any one of them overlapping. There are more sophisticated analysis to show the expected level of overlapping you're likely to get; but it seems safe to say that you could do over a thousand images a day before getting overloaded.

Now the question seems easier: Do you think you'll get more than a thousand or two image creations a day anytime soon? If so, then set a queue; if not, leave it for later.

In any case, keep good logs; make sure that you could tell when there's any processing overlap. Remember that once you get two tasks processing concurrently, they'll take longer, increasing probabilities that a third one could arrive before finishing the other two, and a fourth... when you arrive to an invisible threshold, performance will plummet drastically. Don't lose sleep on this, just don't let it happen before you notice.