Sometimes my whole Django based site goes into 504 gateway timeout errors so none page can be displayed.
Is it possible to write a shell program and cronjob it run every 5 minutes to detect such errors and restart FastCGI process if needed?
I'm currently using command below to restart FastCGI in case it's crashed but it doesn't work for above situation because when 504 gateway errors occur, the processes are still running.
ps ax | grep -v grep | grep port=8001 > /dev/null || restart_fcgi.sh
Perhaps you should find out why the site starts serving 504 errors first - check the logs (or add logging if you don't have enough information), and fix whatever the problem is. Alternatively, you may find nginx + apache + mod_wsgi a more stable approach - nginx is a fantastic front-end webserver for serving static content, but Apache is excellent for hosting dynamic processes. Try to combine them both. I used to use nginx + fastcgi, but I found the former to be a much more stable approach.
If you are still unable to determine what's causing your problem, you can set up a monitor script that runs locally and checks the response code - if it finds a 5XX it can force restart your fastcgi process. You can use Python + httplib for the script, and os.system running your shell script (plus a suitable kill -9 statement) above.
I think if you are having this problem, then supervisor is your friend. A very good friend. Like daemontools, except easy config , lots of help and non root usage.
You should try to find the problem for sure, but regardless, an hour of prep means that you'll be able to rely on the process restarting after failure for long enough to live your life.