What is the cause and how to fix 503 errors with this in Apache error_log: "Broken pipe: ajp_ilink_send(): send failed"

Question

I'm having intermittent problems with a servlet running on JBoss, with Apache forwarding it all requests via mod_proxy_ajp.so.

Sometimes, for REST requests, I get 503 errors from Apache. When this happens, the Apache error_log has this in it:

[Mon Oct 12 09:10:19 2009] [error] (32)Broken pipe: ajp_ilink_send(): send failed
[Mon Oct 12 09:10:19 2009] [error] (32)Broken pipe: proxy: send failed to 127.0.0.1:8009 (localhost)

After a few failed attempts, it starts working again.

I've googled some and found that I'm not the only one who has encountered the problem. The only solution I've found is to make sure that Apache is started after JBoss (I restart Apache after restarting JBoss).

The strange thing about this problem is that there are other servlets running in this JBoss and I don't have the problem there.

The servlet is CXF JAX-RS based.

Apache is 2.2.6.

Answer 1

Broken pipe: send failed messages generally mean that the connection was closed by the other side.

Are there any exceptions in your JBoss logs at the same time? Maybe something is being logged at DEBUG or TRACE level that could provide a hint.

Answer 2

When using the AJP protocol, you have to be very careful to make sure that both sides of the communication (i.e. Apache and Tomcat) are configured with the same parameters. This is because AJP uses persistent, stateful connections, and both parties need to have the same expectations of the connection lifecycle.

I suggest giving the relevant Tomcat docs a good read. You'll probably have to modify either Apache's mod_proxy_ajp config, or Tomcat's AJP connector config, or both, so that they match. If the configs are even slightly different, AJP's performance can absolutely suck.

Answer 3

I've experienced the same problem, but haven't found the cause either. An easy solution is to dump mod_proxy_ajp in favor of mod_proxy_http, if the slight performance penalty is acceptable. Works without problems at least for a website with max ~100 pageloads per second.

Answer 4

I've found that this config generator is helpful when configuring AJP connections. Starting with the generated config and reading the relevant documentation was instructive.

(You can determine the "Apache mpm" parameter by executing apachectl -l, which lists compiled-in modules.)