Why am I getting an Apache Proxy 503 error?

Question

My server was doing just fine up until yesterday. It was running Redmine (a ruby development tracking application), and it was the happiest little server until my "friend" imported a sql table that my little guy couldn't take. Unfortunately after an hour of trying to get the lil guy to respond, we had to power cycle him.

Now after restart, we get a 503 error when trying to visit the domain connected to Redmine. It's hooked up to a mongrel daemon, and we use Apache Proxy to direct all connections to the port Redmine is running on.

Using Lynx on the server (http://localhost:8000) you can see the happy ruby application working fine. But this bit is not working in my apache conf:

<VirtualHost *:80>
    ServerName sub.example.com
    ProxyPass / http://localhost:8000
    ProxyPassReverse / http://localhost:8000
    ProxyPreserveHost on
    LogLevel debug
</VirtualHost>

Here's the error log output for Apache:

[debug] mod_proxy_http.c(54): proxy: HTTP: canonicalising URL //localhost:8000
[debug] proxy_util.c(1335): [client 216.27.137.51] proxy: http: found worker http://localhost:8000 for http://localhost:8000/
[debug] mod_proxy.c(756): Running scheme http handler (attempt 0)
[debug] mod_proxy_http.c(1687): proxy: HTTP: serving URL http://localhost:8000/
[debug] proxy_util.c(1755): proxy: HTTP: has acquired connection for (localhost)
[debug] proxy_util.c(1815): proxy: connecting http://localhost:8000/ to localhost:8000
[debug] proxy_util.c(1908): proxy: connected / to localhost:8000
[debug] proxy_util.c(2002): proxy: HTTP: fam 2 socket created to connect to localhost

[error] (13)Permission denied: proxy: HTTP: attempt to connect to 127.0.0.1:8000 (localhost) failed
[error] ap_proxy_connect_backend disabling worker for (localhost)

[debug] proxy_util.c(1773): proxy: HTTP: has released connection for (localhost)

Answer 1

Are you sure they're restarting in the correct order? I've had weird issues where Apache starts, then Mongrel starts and although Mongrel is running, Apache still throws the proxy error.

I've solved this in the past with various incantations and restarts of Apache and eventually the gods are happy. It seems that sometimes the Mongrel processes don't properly shut down so you have to manually kill them. Here's a link with some [possible] help.

I ended up adding a "kill" option to my /etc/init.d/ mongrel script because it happened so much. It stop Mongrel, killed all Mongrel sessions, started Mongrel and restarted Apache.

<snip>
    kill)
      echo "Stopping, killing, starting, and restarting Apache..."
      mongrel_cluster_ctl stop -c $CONF_DIR --clean
      killall -u mongrel
      mongrel_cluster_ctl start -c $CONF_DIR --clean
      /etc/init.d/httpd restart
      RETVAL=$?
  ;;
</snip>

Probably not a very good solution but the evil went away.

Answer 2

In order to get this to work, I needed to turn of SELinux, and then add trailing slashes to the "localhost:8000" lines.

Here's the code to turn off SELinux:

echo 0 >/selinux/enforce

Answer 3

Try running monit to monitor your mongrels behind Apache, and that way it can restart mongrels for you if they die or get too hungry for memory. If for any reason Apache still gets confused you may just have to gracefully restart apache and it should resolve itself, but for 99% of cases having monit watch over your mongrels should avoid this happening again. The other option is look into Phusion Passenger.