(See this question in ServerFault)
I have a Java client that uses Socket to open concurrent connections to the same machine. I am witnessing a phenomenon where one request completes extremely fast, but the others see a delay of 100-3000 milliseconds. Packet inspection using Wireshark shows all SYN packets beyond the first wait a long time before leaving the client. I am seeing this on both Windows and Linux clients. What could be causing this? This happens when the client is a Windows 2008 or a Linux box.
Code attached:
import java.util.*;
import java.net.*;
public class Tester {
public static void main(String[] args) throws Exception {
if (args.length < 3) {
usage();
return;
}
final int n = Integer.parseInt(args[0]);
final String ip = args[1];
final int port = Integer.parseInt(args[2]);
ExecutorService executor = Executors.newFixedThreadPool(n);
ArrayList<Callable<Long>> tasks = new ArrayList<Callable<Long>>();
for (int i = 0; i < n; ++i)
tasks.add(new Callable<Long>() {
public Long call() {
Date before = new Date();
try {
Socket socket = new Socket();
socket.connect(new InetSocketAddress(ip, port));
}
catch (Throwable e) {
e.printStackTrace();
}
Date after = new Date();
return after.getTime() - before.getTime();
}
});
System.out.println("Invoking");
List<Future<Long>> results = executor.invokeAll(tasks);
System.out.println("Invoked");
for (Future<Long> future : results) {
System.out.println(future.get());
}
executor.shutdown();
}
private static void usage() {
System.out.println("Usage: prog <threads> <url/IP Port>");
System.out.println("Examples:");
System.out.println(" prog tcp 10 127.0.0.1 2000");
}
}
Update - the problem reproduces consistently if I clear the relevant ARP entry before running the test program. I've tried tuning the TCP retransmission timeout, but that didn't help. Also, we ported this program to .Net, but the problem still happens.
Updated 2 - 3 seconds is the specified delay in creating new connections, from RFC 1122. I still don't fully understand why there is a retransmission here, it should be handled by the MAC layer. Also, we reproduced the problem using netcat, so it has nothing to do with java.