How do I handle blocking IO in mod_wsgi/django?

Question

I am running Django under Apache+mod_wsgi in daemon mode with the following config:

WSGIDaemonProcess myserver processes=2 threads=15

My application does some IO on the backend, which could take several seconds.

def my_django_view:
    content=... # Do some processing on backend file
    return HttpResponse(content)

It appears that if I am processing more than 2 http requests that are handling this kind of IO, Django will simply block until one of the previous requests completes.

Is this expected behavior? Shouldn't threading help alleviate this i.e. shouldn't I be able to process up to 15 separate requests for a given WSGI process, before I see this kind of wait?

Or am I missing something here?

Answer 1

If the processing is in python, then Global Interpreter Lock is not being released -- in a single python process only one thread of python code can be executing at a time. The GIL is usually released inside C code though -- like most I/O, for example.

If this kind of processing is going to happen a lot, you might consider running a second "worker" application as a deamon, reading tasks from the database, performing the operations and writing resulsts back to the database. Apache might decide to kill processes that take too long to respond.

Answer 2

+1 to Radomir Dopieralski's answer.

If the task takes long you should delegate it to a process outside the request-response cycle, either by using a standard cron, or some distributed task queue like Celery