Java Linux Nonblocking Socket Timeout Behavior

Question

I have a Java nonblocking server that keeps track of all the socket channels in a selector. I then establish 500 connections to the server and send data regularly. Every piece of data the server receives is echoed back to the client.

The problem comes where the test works wonderfully for a couple of hours and then all of the sudden gradually all of the sockets the server is managing throw a Connection timed out IOException when attempting to read data.

I've looked into whether or not the client thread was being starved (and not sending data), but I am yielding to the client thread that iterates through all the sockets and writes out data. Traffic seems to be constantly flowing properly, but after a while it just all dies out. Any ideas what could be causing this behavior?

I'm running on a Linux platform with the latest iteration of Java 6. My application launches two threads, one for the server, and one for all the clients. Thanks in advance!

Extra: The issue is dealing with Linux and not my code. When I run the exact same setup on a Windows box (on the same hardware) it never times out, but after several hours they start to occur on Linux. It must be some kind of TCP setting in Linux that's causing it to happen. Thanks for the suggestion.

Answer 1

The -doCloseWithReadPending option in Java and JRE versions 1.5 or 5.0 allows one thread to close a socket when there is a read pending on the same socket from another thread.

When close() is called on a socket which has an outstanding read call from another thread, the close() by default blocks the socket until the read call completes.

With the -doCloseWithReadPending option, the socket close() call closes the socket and in the context of the thread with the pending read, a SocketException with the message "Socket closed" is thrown.

I don't know if this is the root cause of your issue without seeing the code, but I thought I would add this here incase it affects your issue.

Answer 2

The issue is dealing with Linux and not my code. When I run the exact same setup on a Windows box (on the same hardware) it never times out, but after several hours they start to occur on Linux. It must be some kind of TCP setting in Linux that's causing it to happen. Thanks for the suggestion.

Answer 3

So in both the case that works (Windows with recent JVM) and the case that doesn't (Linux with recent JVM), both the server and client are on the same machine in the same JVM?

Can you clarify what "all of the sudden gradually" means? Like, after a few hours -- and always the same number of hours -- then within a few seconds all server-side sockets throw exceptions?

You don't mention the client thread reading the data that comes back. Perhaps it's stopped and you haven't noticed. (What is the client thread doing when the server thread encounters the 500 rapid exceptions? Try a few stack dumps in succession to see.)