apache2 ignores MaxKeepAliveRequests, closes connections arbitrarily

Question

We have a Tomcat front-end server that proxies to our Apache 2.2.11 app server, running on a 64-bit Fedora 2.6.21.7 EC2 2xlarge instance (AKI aki-b51cf9dc). Apache is running mod_perl and is not threaded.

We are trying to have the connections persist for a long time between Tomcat, running on another EC2 instance, and the Apache server, while not persisting connections from outside clients coming directly into the Apache server. Our config looks like this:

Listen 80
NameVirtualHost *:80

# used for external clients
<VirtualHost *:80>
    ServerName xxx.yyy.com
    ServerAlias *.yyy.com
    DocumentRoot "/var/www/html"
    KeepAlive Off
</VirtualHost>

# used from tomcat server on local network
<VirtualHost *:80>
    ServerName ip-<Apache-server-local-IP>.ec2.internal
    KeepAlive On
    KeepAliveTimeout 3600
    MaxKeepAliveRequests 0
    DocumentRoot "/var/www/html"
</VirtualHost>

TimeOut 60
MinSpareServers 20
MaxSpareServers 30
StartServers 20
MaxClients 60
GracefulShutdownTimeout 90

We've tried all sorts of values for MaxKeepAliveRequests and KeepAliveTimeout, and the server definitely maintains a connection for a while with Tomcat, but it always closes it within a matter of seconds, when only some tens of requests have been processed. It may be significant that I've never seen a process maintain 100 or more connections on a socket while observing using mod_status.

There are never any persistent connections with the non-Tomcat clients, so we know that there is some difference going on there and the VirtualHost config is definitely being applied in both cases.

I should mention that the requests from Tomcat are all POSTs while the others are a mixture of POSTs and GETs.

When I look a the traffic on a given port with tcpdump I can clearly see a number of POSTs being processed correctly and then at some point after returning a good reply (200, data looks fine) the Apache server immediately closes the connection, sending a FIN to Tomcat. This happens in cases where there is absolutely no difference between the last and second-to-last requests or replies other than minor data like the real client's IP, so there's no indication of the server barfing while processing a request. And of course there's nothing suspicious in the error logs and the httpd processes themselves are not dying.

From netstat we can see the connections to the Tomcat server being held open for some seconds, but cycling pretty quickly through the range of remote ports, verifying what we see elsewhere. It's almost like Apache is trying to fairly allocate connections to prevent the persistent ones from starving the others--but it wouldn't do that, would it?!

I'd like nothing more than to be told that we're doing something dumb here! Please, please tell me I'm an idiot, or at least near-sighted...

Answer 1

What is the value of cat /proc/sys/net/ipv4/tcp_keepalive_time on the Tomcat ?

Is it unusually low? Default is 7200 (i.e. 2 hours)

Answer 2

Thanks for MaxKeepAliveRequests in apache tutorial.