How to retain one million simultaneous TCP connections?

tags:

server
tcp

views:

209

answers:

+9 Q:

How to retain one million simultaneous TCP connections?

I am to design a server that needs to serve millions of clients that are simultaneously connected with the server via TCP.

The data traffic between the server and the clients will be sparse, so bandwidth issues can be ignored.

One important requirement is that whenever the server needs to send data to any client it should use the existing TCP connection instead of opening a new connection toward the client (because the client may be behind a firewall).

Does anybody know how to do this, and what hardware/software is needed (at the least cost)?

Thanks in advance for any suggestions.

+4 A:

This problem is related to the so-called C10K problem. The C10K page lists a large number of good resources for addressing the problems you will encounter when you try to allow thousands of clients to connect to the same server.

Greg Hewgill 2010-05-14 02:13:39

thanks for the answer. but personally i don't think they are the same problem. what i want to know is how to keep 1M connected clients persistently connected, NOT how to accept their connection requests or how to detect their connection status change.thanks anyway.

cow 2010-05-14 03:14:15

@cow: There is nothing particularly hard about keeping the clients connected - what makes you think there would be? It's far from the most challenging part of the problem.

caf 2010-05-14 03:22:39

what if the clients are within a network where their IPs may be frequently changed. For example, i have a T-Mobile G1 phone. i found IP of my phone frequently changes. even if the phone has a TCP connection to some server outside T-Moble network, when there is no data flowing via the connection, the chance that IP of the phone will be changed is big; once IP is changed, any TCP connection is in fact broken. that is why i have the problem.

cow 2010-05-14 03:34:10

as i know, Cisco 7600 router support up to (only) 64k TCP connections and costs ~$50k. do i need 20 Cisco 7600 routers, tens of linux servers and probably several load balancer to support 1M simultanous connected users?

cow 2010-05-14 03:34:59

With a typical TCP/IP server design, you're limited in the number of simultaneous open connections you can have. The server has one listening port, and when a client connects to it the server makes an accept call, and that creates a new socket on a random port for the rest of the connection.

To handle more than 64K simultaneous connections I think you need to use UDP instead. You only need one port for the server to listen on, and you need to manage the connections using a 32-bit client ID in the packet data instead of having a separate port for each client. The 32-bit client ID could be the client's IP address, and the client can listen on a known UDP port for messages coming back from the server. That port would be the only one that needs to be open on the firewall.

With this approach, your only limitation is how quickly you can handle and respond to UDP messages. With millions of clients, even sparse traffic could give you large spikes, and if you don't read the packets fast enough your input queue will fill up and you'll start dropping packets. The C10K page Greg points to will give you strategies for that.

DougWebb 2010-05-14 02:31:26

Multiple client connections to a server do not use additional *server* ports. There will still be technical limitations, but it's not the number of ports. Connections are identified on the server side using the unique 4-tuple (server_ip, server_port, client_ip, client_port). You may be thinking of the *socket handles*, where a server socket appears to spawn more socket handles through the `accept()` call.

Greg Hewgill 2010-05-14 02:40:47

Hmm. Yeah, I guess you're right. http://linux.die.net/man/2/accept says accept will fail if you run out of file descriptors, not ports. I took a look in the linux kernel source code, and from what I can tell the maximum number of files is passed around as an int. That probably limits you to 32K file descriptors, though it'll vary by platform. I'm wrong on the details of why TCP is limited, but I think it's still limited and UDP is a workable alternative.

DougWebb 2010-05-14 03:09:49

thanks for comments from you both.i saw the c10k page weeks ago. i remember it suggest epoll(). i wish i could use it with UDP.howver, as i know, epoll is irrelevant to a UDP server that has only one socket to receive, am i correct?the urgency to read packets fast enough is valid. however, i cannot think of a better way to read faster than using a dedicate thread to read packets off the UDP socket and put them in a queue for worker threads to consume. could you advise better methods? many thanks in advance.

cow 2010-05-14 03:23:47

This isn't correct. The connection, at least on Windows operating systems, is identified by a tuple of host address, host port, remote address, remote port. Thus whilst you may be limited by port numbers for the number of OUTBOUND connections (as you only have a single range of ephemeral ports to select from) you are not limited for inbound connections from different hosts. I've run tests on a low spec Windows Server 2003 machine, here: http://www.lenholgate.com/archives/000568.html, that easily achieve more than 70,000 concurrent inbound TCP connections.

Len Holgate 2010-05-14 05:34:35

@cow: If you need to do work with each packet as they come in, spinning them into a queue for worker threads to consume sounds right. You'll have to balance the number of workers, how much work they do, and the incoming packet rate, and make sure your queue can grow to handle backlogs. I had a similar project with bursty incoming traffic, and I wrote the packets out to a file for batch processing later during quiet periods.

DougWebb 2010-05-21 21:43:09

+1 A:

I've come across the APE Project a while back. It seems like a dream come true. They can support up to 100k concurrent clients on a single node. Spread them across 10 or 20 nodes, and you can serve millions. Perfect for RESTful applications. Might want to look deeper for any shared namespace. One drawback is that this is a standalone server, as in supplementary to a web server. This server is of course Open Source, so any cost is hardware/ISP related.

Vic 2010-05-14 03:12:54

thanks for the information.but my app is not web based. how can i utilize APE?

cow 2010-05-14 03:46:03

You can use APE without a browser. They employ a default protocol (and you may create your own: http://www.ape-project.org/wiki/index.php/Protocol) You can write your own library to manage connections with your language of choice using their "APE Javascript framework" as a reference.

Vic 2010-05-14 03:56:16

thanks again. i will look into the protocol to see if it is good to replace my own binary-encoded protocol with APE's.

cow 2010-05-14 04:15:00

+3 A:

What operating systems are you considering for this?

If using a Windows OS and using something later than Vista then you shouldn't have a problem with many thousands of connections on a single machine. I've run tests (here: http://www.lenholgate.com/archives/000568.html) with a low spec Windows Server 2003 machine and easily achieved more than 70,000 active TCP connections. Some of the resource limits that affect the number of connections possible have been lifted considerably on Vista (see here: http://www.lenholgate.com/archives/000837.html) and so you could probably achieve your goal with a small cluster of machines. I don't know what you'd need in front of those to route the connections.

Windows provides a facility called I/O Completion Ports (see: http://msdn.microsoft.com/en-us/magazine/cc302334.aspx) which allow you to service many thousands of concurrent connections with very few threads (I was running tests yesterday with 5000 connections saturating a link to a server with 2 threads to process the I/O...). Thus the basic architecture is very scalable.

If you want to run some tests then I have some freely available tools on my blog that allow you to thrash a simple echo server using many thousands of connections (1) and (2) and some free code which you could use to get you started (3)

The second part of your question, from your comments, is more tricky. If the client's IP address keeps changing and there's nothing between you and them that is providing NAT to give you a consistent IP address then their connections will, no doubt, be terminated and need to be re-established. If the clients detect this connection tear down when their IP address changes then they can reconnect to the server, if they can't then I would suggest that the clients need to poll the server every so often so that they can detect the connection loss and reconnect. There's nothing the server can do here as it can't predict the new IP address and it will discover that the old connection has failed when it tries to send data.

And remember, your problems are only just beginning once you get your system to scale to this level...

Len Holgate 2010-05-14 05:48:24

ansaurus

tags:

views:

answers:

How to retain one million simultaneous TCP connections?

related questions