How do I open 20000 clients in Java without increasing file limit?

Question

Whenever I open a socket channel. If the client accepts then 1 file descriptor is created internally so I can create a maximum of 1024 clients in Linux.
But I want to create more clients without increasing file descriptor limit in Linux (ulimit -n 20000) So how can I create more sockets in Java?

Answer 1

If your session is limited to 1024 file descriptors you can't use more then that from a single JVM.

But since the ulimit is a per-process limitation, you could probably get around it by starting more JVMs (i.e. to get 2048 connections start two JVMs each using 1024).

Answer 2

If you are using UDP, can you multiplex on a single local socket youself? You'll be able to separate incoming packets by their source address and port.

If it's TCP you're out of luck, and the TIME_WAIT period after closing each socket will make things worse.

Answer 3

If you really are looking at coping with a huge number of connections then the bast way to do it scalably would be to implement a lightweight dataserver process that has no responsibility other than accepting and forwarding data to a parent process.

That way as the each dataserver gets saturated you simply spawn a new instance to give yourself another 1024 connections. You could even have them exist on seperate machines if needed.

Answer 4

Why cant you increase the ulimit ? It seems like an artificial limitation. There is no way from java code (afaik) that allows you access to the system to reset the ulimit - it needs to be set before the process starts - in a startup script or something similar.

The JBoss startup scripts peform a 'ulimit -n $MAX_FD' before they start Jboss ...

Len

Answer 5

The limit RLIMIT_NOFILE is enforced by the operative system and limits the highest fd a process can create. One fd is used for every file, pipe and socket that is opened.

There are hard and soft limits. Any process (like your shell or jvm) is permitted to change the soft value but only a privileged process (like a shell run by the root user) can change the hard value .

a) If you are not permitted to change the limit on the machine, find someone that are.

b) If you for some reason can't be bothered to type ulimit, I guess you can call the underlying system call using JNA : man setrlimit(2). (.exec() won't do as it's a built in command)

See also Working With Ulimit

Answer 6

We recently upped our ulimit because our java process was throwing lots of "Too many files open" exceptions.

It is now 65536 and we have not had any issues.