views:

240

answers:

9

I'm writing a Java server which uses plain sockets to accept connections from clients. I'm using the fairly simple model where each connection has its own thread reading from it in blocking mode. Pseudo code:

handshake();

while(!closed) {
  length = readHeader(); // this usually blocks a few seconds
  readMessage(length);
}

cleanup();

(Threads are created from an Executors.newCachedThreadPool() so there shouldn't be any significant overhead in starting them)

I know this is a bit of a naive setup and it wouldn't really scale well to many connections if the threads were dedicated OS threads. However, I've heard that multiple threads in Java can share one hardware thread. Is that true?

Knowing that I'll be using the Hotspot VM on Linux, on a server with 8 cores and 12GB of RAM, do you think this setup will work well for thousands of connections? If not, what are the alternatives?

+3  A: 

It is possible this will scale to thousands of clients. But how many thousands is the next question.

A common alternative is to use Selectors and non-blocking I/O found in the java.nio package.

Eventually you get into the question of whether it's useful to set up your server in a clustered configuration, balancing the load over multiple physical machines.

Mark Peters
+4  A: 

This will scale well for up to hundreds of connections, not to thousands. One issue is that a Java thread takes quite a bit of stack as well (e.g. 256K), and the OS will have problems scheduling all your threads.

Look at Java NIO or framworks that will help you get started doing complex stuff more easily (e.g. Apache Mina)

ivy
Each thread will take some stack, but if I transform it into a non-blocking model each connection will need more data on the heap, such as "in what phase of reading a message are we now", which is currently simply determined by the instruction pointer (correct word?) of a thread. The scheduling might be an issue though.
Bart van Heukelom
@Bart: The extra space per connection is nowhere near as large as a stack; going to NIO will improve scalability. The cost is that most folk find it harder to grok what's going on (and Java doesn't have coroutines, which can be used to balance the concerns).
Donal Fellows
As ivy said, the main consideration to scaling to thousands is really the stack they will end up consuming. On the other hand, I am not fully convinced that scheduling will be a major factor. It will be some factor, but perhaps not as great as one might perceive. If your dominant activities are I/O, then your threads will promptly yield the CPUs in most cases, and they will get paged in only if data is received. If you think about it, that picture is not too different even if you use NIO. It's just that more number of threads are involved in asking for roughly the same number of CPU cycles.
sjlee
+3  A: 

To have a good perfomance when handling many sockets you usually use a select approach that is how Unix API handles single-threaded multi-socket applications that need many resources.

This can be done through the java.nio package that has a Selector class which basically is able to go through all the opened sockets and notify you when new data is available.

You register all the opened streams inside a single Selector and then you can handle all of them from just one thread.

You can get additional infos with a tutorial here

Jack
By accident I stumbled upon a SO question about the method `InputStream.available()`, which I'd forgotten about. Could you tell me the advantage of using a `Selector` as opposed to having a thread handle multiple connections, using `available()` to prevent blocking?
Bart van Heukelom
Because you avoid to reimplement something that goes over the various Sockets programmatically since you will be using something specifically tailored for this purpose :) The main advantage would be saving time for debug and implementation..
Jack
Hmm, but the implementation of that, at least in my case, would be extremely simple. My protocol is 1. read integer that specifies message length. 2. read that number of bytes in one go
Bart van Heukelom
In that case it's a matter of preference. Of course avoiding to use `nio` when you don't know it neither you need it would be overkill but if you plan to have many sockets why don't give channels and asynchronous IO a chance? Mind that you will have to care about interrupting the thread to avoid having it going in an infinite loop when data isn't ready..
Jack
+1  A: 

Threads aren't as expensive as they used to be, so an "ordinary" IO implementation can be ok to a point. However if you are looking at scaling to thousands or beyond it is probably worth investigating something more sophisticated.

The java.nio package solves this by providing socket multiplexing/non-blocking IO which allows you bind several connections to one Selector. However this solution is much harder to get right than the simple blocking approach because of the multithreading and non-blocking aspect.

If you wish to pursue something beyond the simple IO then I would suggest looking at one of the good quality network abstraction libraries out there. From personal experience I can recommend Netty which does most of the fiddly NIO handling for you. It does however have a bit of a learning curve but once you get used to the event based approach it is very powerful.

Mike Q
+1  A: 

If you have any interest in leveraging deployment and management of an existing container, you might look at making a new protocol handler inside of Tomcat. See this answer to a related question.

UPDATE: This post from Matthew Schmidt claims the NIO-based connector (written by Filip Hanik) in Tomcat 6 achieved 16,000 concurrent connections.

If you want to write your own connector, take a look at MINA to help with NIO abstractions. MINA also has management features which may eliminate need for another container (should you be concerned about deployment of many units and their operation, etc.)

unhillbilly
Interesting, but my application is not currently using a container. I'm also interesting in working with NIO myself (if it turns out to be necessary) for education purposes.
Bart van Heukelom
+2  A: 

The JVM for Linux is using one to one thread mapping. This means that every Java thread is mapped to one native OS thread.

So creating a thousand of threads or more is not a good idea because it will impact your performance (context switching, cache flushes / misses, synchronization latency etc). It also doesn't make any sense if you have less than a thousand of CPUs.

The only adequate solution for the serving many clients in parallel is to use asynchronous I/O. Please see this answer on Java NIO for details.

See also:

Vlad Lazarenko
Thanks, this is a helpful answer, but I don't agree with "It also doesn't make any sense if you have less than a thousand of CPUs.". It makes sense for keeping the current state of the connection ("are we now reading a header or data", "how much of the message did we receive yet", etc.) on the stack rather than the heap, which is faster (right?) and makes it easier to program.
Bart van Heukelom
It only makes you feel like it is easier to program (both approaches are relatively easy IMO). Using stack doesn't make it faster de facto, it all depends on what you are doing. In any case, trade-off having 1K threads will be greater than maintaining a list of sessions with pre-allocated state.
Vlad Lazarenko
+1 for Green Threads
Tim Bender
+1  A: 

Try Netty.

the "one thread per request" model is the way most Java app servers are written. Your implementation can scale as well as they do.

duffymo
A: 

I think a better approach is to not handle threads yourself. Create a pool (ThreadExecutor or some other stuff) and simple dispatch work to your pool.

Of course, I think asynchronous I/O will make it better and faster, but will help you with socket and networking problems. Only. When your threads block because of I/O, the JVM will put it to sleep and change for another thread until the blocking I/O return. But this will block only the thread. Your processor will continue to run and start to proccess other thread. So, minus the time to create a thread, the way you use I/O not affect not much your model. If you do not create threads (using a pool) your problem is solved.

Plínio Pantaleão
I do use an Executor, but since each thread basically does `while(!closed) read();` that doesn't actually decrease the number of threads, it just diminishes the overhead of creating them.
Bart van Heukelom
Its exactly my point. When you call read() your thread will block until it has something to do. Doesnt matter how many threads you have, but how many RUNNABLE threads you have, and the blocked threads dont count. They will not compete for processor time. So, in the end you will have only the threads that have things to do competing by processor time.
Plínio Pantaleão
+1  A: 

I'd suggest it depends more on exactly what else the server is doing when it processes the messages. If it's relatively lightweight then your machines spec should EASILY cope with merely handling the connections of thousands of such processes. Tens of thousands is another question perhaps, but you only need two machines on the same network to actually empirically test it and get a definite answer.