TCP/IP failed to establish an outgoing connection

Question

We're seeing TCP/IP warning and quite a few connection failures on our web and sql server (Win2k8 SP1 w/ IPv6 enabled) and it happens more often when the worker process/sql server process has been running for a couple of days continously. I am wondering what causes these warnings? What is the default number of tcp connections allowed and how can we bump this up?

TCP/IP failed to establish an outgoing connection because the selected local endpoint was recently used to connect to the same remote endpoint. This error typically occurs when outgoing connections are opened and closed at a high rate, causing all available local ports to be used and forcing TCP/IP to reuse a local port for an outgoing connection. To minimize the risk of data corruption, the TCP/IP standard requires a minimum time period to elapse between successive connections from a given local endpoint to a given remote endpoint.

Answer 1

Here is the TechNet page for that error.

Answer 2

are you using something like

setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(on)

it allows you to reuse the same socket twice without warning

Answer 3

This looks to me like exactly what the error message suggests... your ports are being exhausted. This implies a LOT of outgoing connections.

The default is not to limit the number of tcp connections (I suppose you're only limited by the number of ports, probably ports 1024 to 65534)

I would begin investigating why you're seeing so many outgoing connections in such a short time.

Answer 4

It sounds kinda like your connections start failing and then are automatically retried. If the retries happen quickly enough TCP could cycle through the entire port range and all your ports will be in TIME-WAIT state (IIRC that's what the state's called, it's been a little while since I've been debugging TCP in detail).

A couple of things to understand first:

1. Who is originating the connection(s) and what are they supposed to be used for?
2. Verify the connection failure rates, I tend to use Wireshark for this sort of thing.

Then you have to look for clues as to why the connection is failing. Probably the outgoing connection is to a server, is the server up and running? Why wouldn't the server accept the connection - is the process/thread responsible for accepting the connection deadlocked, blocked on something, or has it exited? Is the network between the originator and the server working normally?