AIX: IBM Java: java.net.SocketException: Connection timed out:could be due to invalid address

Question

We have seen the following exceptions very frequently on IBM AIX when attempting to make an SSL connection to our server:

java.net.SocketException: Socket closed
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(DashoA6275(Compiled Code))
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(DashoA6275(Compiled Code))
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java(Inlined Compiled Code))
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java(Compiled Code))
at java.io.FilterOutputStream.flush(FilterOutputStream.java(Compiled Code))
at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java(Compiled Code))
at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java(Compiled Code))
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java(Compiled Code))
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java(Compiled Code))
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java(Compiled Code))
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java(Compiled Code))
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java(Inlined Compiled Code))
at com.eximtechnologies.httptransport.client.ClientTransport.receiveMessages(ClientTransport.java(Compiled Code))
at com.eximtechnologies.httptransport.client.ClientTransport.receiveMessages(ClientTransport.java(Inlined Compiled Code))
at com.eximtechnologies.ecserver.connection.XMSHTTPConnection.checkForNewMessages(XMSHTTPConnection.java(Compiled Code))
at com.eximtechnologies.ecserver.connection.XMSHTTPConnection.timeoutExpired(XMSHTTPConnection.java(Compiled Code))
at com.eximtechnologies.xmd.timer.TimerEvent$1.run(TimerEvent.java(Compiled Code))

From the error, you would think this was just a network problem, but the client had never experienced the problem before about 2 months ago, and AFAIK, there haven't been any changes to the network layout.

We also receive this fairly frequently:

java.net.SocketException: Connection timed out:could be due to invalid address
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:336)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:201)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:188)
at java.net.Socket.connect(Socket.java:478)
at java.net.Socket.connect(Socket.java:428)
at java.net.Socket.<init>(Socket.java:335)
at java.net.Socket.<init>(Socket.java:210)
at javax.net.ssl.SSLSocket.<init>(Unknown Source)

I'm suspecting that this is an AIX problem, but I guess it could be a firewall issue? I also saw some people in google searches hinting at a problem with commons http, but I couldn't see how that would be related.

Is this something that others have seen with AIX recently?

Answer 1

I have had issues with http client that were corrected by using a multithreaded connection. We fixed it by moving from the first to the second of the configurations below:

<bean id="httpClient" class="org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor">
    <property name="httpClient">
     <bean class="org.apache.commons.httpclient.HttpClient">
      <property name="connectionTimeout"><value>1000</value></property>
      <property name="timeout"><value>3000</value></property>
     </bean>
    </property>
</bean>

<bean id="httpClient" class="org.springframework.remoting.httpinvoker.CommonsHttpInvokerRequestExecutor">
    <property name="httpClient">
     <bean class="org.apache.commons.httpclient.HttpClient">
      <property name="connectionTimeout"><value>1000</value></property>
      <property name="timeout"><value>3000</value></property>
      <property name="httpConnectionManager">
        <bean class="org.apache.commons.httpclient.MultiThreadedHttpConnectionManager" destroy-method="shutdown">
       <property name="params">
         <bean class="org.apache.commons.httpclient.params.HttpConnectionManagerParams">
        <property name="defaultMaxConnectionsPerHost" value="20" />
         </bean>
       </property>
        </bean>
      </property>
     </bean>
    </property>
</bean>

Answer 2

"java.net.SocketException: Socket closed" means that your side closed the socket. You say that this happens when you attempt to make an SSL connection to your server. However, the stack trace suggests that this happens when HTTPClient attempts to write an HTTP request over an already established connection.

This could happen if, for example, you somehow managed to make HTTPClient send a request via a connection that was previously closed by HTTPClient, or, more likely, by some other code on your side. Check whether you are accessing the underlying socket somewhere. Or it could be that the socket is closed by the SSL/TLS protocol (if I'm not mistaken, SSL/TLS has its own higher-level protocol for closing the underlying connection), but HTTPClient somehow managed to not notice this (don't know whether it's possible, but, say, the remote side closed the SSL connection but was using HTTP/1.1 persistent connections and didn't set a Connection: close response).

You can troubleshoot these issues by analyzing the TCP traffic using tcpdump/Wireshark. You could also start an stunnel on a machine to the server's HTTPS port, then make your code communicate with the server over plain HTTP via this tunnel. This should enable you to see the HTTP traffic in plaintext.

"java.net.SocketException: Connection timed out" means that the TCP connection could not be established due to a timeout. Could be that the packets are dropped by a firewall. For example, it could be that you need to use an HTTP proxy to make HTTPS requests. It could also be that the server machine is really busy or the network is really busy. Again, I suggest you try tcpdump/Wireshark to see what's going on at the TCP level.

Answer 3

Steve, thanks a lot man. You saved my day. I've been looking arround for this problem with no luck at all. Http Client uses by default SimpleHttpConnectionManager as connection manager. This manager has a default maximun connection per host of 2 :( That was my problem. Thanks again.