Django + lighttpd + fcgi performance

Question

I am using Django to handle fairly long http post requests and I am wondering if my setup has some limitations when I received many requests at the same time.

lighttpd.conf fcgi:

fastcgi.server = (
  "a.fcgi" => (
    "main" => (
      # Use host / port instead of socket for TCP fastcgi
      "host" => "127.0.0.1",
      "port" => 3033,
      "check-local" => "disable",
      "allow-x-send-file" => "enable"
  ))
)

Django init.d script start section:

start-stop-daemon --start --quiet \
  --pidfile /var/www/tmp/a.pid \
  --chuid www-data --exec /usr/bin/env -- python \
  /var/www/a/manage.py runfcgi \
  host=127.0.0.1 port=3033 pidfile=/var/www/tmp/a.pid

Starting Django using the script above results in a multi-threaded Django server:

www-data   342  7873  0 04:58 ?        00:01:04 python /var/www/a/manage.py runfcgi host=127.0.0.1 port=3033 pidfile=/var/www/tmp/a.pid
www-data   343  7873  0 04:58 ?        00:01:15 python /var/www/a/manage.py runfcgi host=127.0.0.1 port=3033 pidfile=/var/www/tmp/a.pid
www-data   378  7873  0 Feb14 ?        00:04:45 python /var/www/a/manage.py runfcgi host=127.0.0.1 port=3033 pidfile=/var/www/tmp/a.pid
www-data   382  7873  0 Feb12 ?        00:14:53 python /var/www/a/manage.py runfcgi host=127.0.0.1 port=3033 pidfile=/var/www/tmp/a.pid
www-data   386  7873  0 Feb12 ?        00:12:49 python /var/www/a/manage.py runfcgi host=127.0.0.1 port=3033 pidfile=/var/www/tmp/a.pid
www-data  7873     1  0 Feb12 ?        00:00:24 python /var/www/a/manage.py runfcgi host=127.0.0.1 port=3033 pidfile=/var/www/tmp/a.pid

In lighttpd error.log, I do see load = 10 which shows I am getting many requests at the same time, this happens few times a day:

2010-02-16 05:17:17: (mod_fastcgi.c.2979) got proc: pid: 0 socket: tcp:127.0.0.1:3033 load: 10

Is my setup correct to handle many long http post requests (can last few minutes each) at the same time ?

Answer 1

I think you may want to configure your fastcgi worker to run multi-processed, or multi-threaded.

From manage.py runfcgi help:

method=IMPL          prefork or threaded (default prefork)
[...]
maxspare=NUMBER      max number of spare processes / threads
minspare=NUMBER      min number of spare processes / threads.
maxchildren=NUMBER   hard limit number of processes / threads

So your start command would be:

start-stop-daemon --start --quiet \
  --pidfile /var/www/tmp/a.pid \
  --chuid www-data --exec /usr/bin/env -- python \
  /var/www/a/manage.py runfcgi \
  host=127.0.0.1 port=3033 pidfile=/var/www/tmp/a.pid \
  method=prefork maxspare=4 minspare=4 maxchildren=8

You will want to adjust the number of processes as needed. Note that the more FCGI processes you have, your memory usage will increase linearly. Also, if your processes are CPU-bound, having more processes than number of available CPU cores won't help much for concurrency.