Java socketserver: How to handle many incoming connections?

Question

I am writing a simple multithreaded socketserver and I am wondering how best to handle incoming connections:

1. create a new thread for each new connection. The number of concurrent threads would be limited and waiting connections limited by specifying a backlog

2. add all incoming connections into a queue and have a pool of worker threads that process the queue

I am inclined to go for option 2 because I really don't want to refuse any connections, even under high loads, but I am wondering if there are any considerations I should be aware of with accepting effectively unlimited connections?

Answer 1

With unlimited connections you potentially can create a huge number of threads. That means a lot of processing is required, plus each thread will consume a fixed amount of memory by default merely for the heap (I think the figure is 512kB per thread, but that may be platform dependent).

By pooling a fixed number of threads and accepting a limited number of clients you will ensure that some of your clients will be serviced in a reasonable time period, and your server won't collapse from overloading.

You may want to check out this article on building servers using NIO or perhaps check out frameworks such as Apache Mina.

Answer 2

If you accept unlimited connections, and your processing isn't fast enough to cope with the stream of incoming connections, your queue will fill up - until it reaches a point, where it takes so long to process previous requests, that most requests won't even be interested in the answer anymore, as soon as you get to them.

Answer 3

1 Thread per connection does not scale. Read about it at The C10K problem. It is a long read, but very famous.

I would advise to learn about NIO or rather use the impressive netty framework which handles all the heavy-lifting(NIO) for you.

Answer 4

I really don't want to refuse any connections

Actually you probably do. When you're overloaded, you want to retain enough capacity to get rid of the current load before you accept any more. Slowing everdybody to a halt isn't any more acceptable than refusing connections.

Queueing theory says that the sweet spot is about 70% utilization. If your server is going to have a steady load higher than that get faster hardware until it doesn't.

Having said that, if you're expecting hundreds of thousands of connections I would use a thread pool or NIO. If you're only expecting thousands, a thread per connection is much the easiest way to go.