The application requester cannot establish the connection. (Too many open files)

Question

I develop application run in Websphere work manager. work manager is used to run thread in the webpshere applications erver.

Every 5 minutes my thread try to get some data from MySQL database from the different host from the application server machine.

When the Host of MySql database turned off, The work manager always try to connect to MySQL database and I know my program will always get exception connection failure. this is the exception: com.mysql.jdbc.CommunicationsException:

Communications link failure due to underlying exception

But, over time my program get exception as follows:

java.sql.SQLException: The application requester cannot establish the connection. (Too many open files)

and this exception make my application server crash:

[8/2/10 9:07:21:613 ICT] 00000d54 prefs         W   Could not lock User prefs. Unix error code 24.
[8/2/10 9:07:21:613 ICT] 00000d54 prefs         W   Couldn't flush user prefs: java.util.prefs.BackingStoreException: Couldn't get file lock.

I need suggestion how to fix this problem and prevent my application being crash ????

WorkEnvironment:

Operation System AIX
Application Server Webpshere 7.0

Answer 1

I had faced similar problem. I fixed it by increasing ulimit of my user on OS. This occurs because there’s a limit to the number of open files in most of the operating systems.

On linux box you can do it by following command. Here I have set it to unlimited.

ulimit -u unlimited

You can also check current limits by running :

>ulimit -a       
      core file size (blocks)   1000000
      data seg size (kbytes)    unlimited
      file size (blocks)            unlimited
      max memory size (kbytes)  unlimited
      stack size (kbytes)           8192
      cpu time (seconds)            unlimited
      max user processes            unlimited  
      pipe size (512 bytes)         8
      open files                    1024
      virtual memory (kbytes)   2105343

Answer 2

Generally speaking, when you connect to a database, you have some sort of open call to open the connection, then you do some work with the connection, and then you close the connection. If you've forgotten to close the connection, you can run out of resources quickly and get an error like the one you're seeing. However, even if you did remember to close the connection, you might hit an exception while doing the work that would cause flow of execution to bypass the close call. For that reason, you always want the work to be wrapped in a try block, and the close call to be in a finally block. Could that be your problem?

Answer 3

It sounds like you have a file descriptor leak.

Some part of your code (or some other code running on the machine) is creating more and more file handles, including sockets, and is not closing them. Based on your description, it sounds like it's your code that's doing this.

I suspect that when you create the socket, you're not closing it cleanly when an exception is thrown. If you don't do this, then the socket will stay open and over time you'll run out of files. Any resource that needs to be closed after use should always be closed in a try-finally block, to ensure that the resource will be closed regardless of the path through the method.

If you don't think you're leaking files, use the lsof utility on the host to see what file handles are being held open by your process, and check that you do legitimately need all of them. I find it unlikely that you have a legitimate reason to exceed the default FD limit of the system.