fetching connection from JNDI >15 minutes!

Question

I have this weird problem, web application connecting to AS400 DB2 server through JNDI, getting connection from JNDI happens to last for about 930-960 seconds!!! Usually it takes 4ms to get a connection, and from time to time it spans to 15 minutes... it has no special rule of when/why it happens. We are using JTOpen jt400.jar driver version 7.0.

We have set it up to have minimum of 20 idle connections in pool, 200 max number of connections in pool, and we use to have 10 people working on web application at once, so there is little chance of utilizing more of 10 connections at once..

Answer 1

Enable garbage collection statistics reporting on your application.

My guess is that the JNDI lookup is big and triggers a "stop-the-world" garbage collection which takes ages for big programs in small pools.

A second guess is that your connections are not returned properly to the connection pool so the JNDI lookup needs to WAIT for a connection to grow old and be discarded and replaced with a new one which your application then gets.

Answer 2

Make sure you always properly close your connections when you don't need them anymore. Otherwise, you will keep allocating more and more connections and eventually, the DB2 server will stop giving you more connections. It will the wait for a couple of minutes for an existing connection to time out and give you that one.

To see how many connections you have, get the excellent Advanced Query Tool which has a whole slew of views to monitor the state of your DB (who is connected, what are they doing, which queries hang in deadlocks, etc). Yes, it costs money but if it saves you one single day of searching a bug, it will already be worth it.

Answer 3

I would suggest couple of things to look at to try to diagnose the issue:

1. First thing to do is to monitor the state of connection pool during this lengthy lookups. This will allow you to find out whether problem is in actual retrieval of connections. Regardless of whether you properly close connections or not timing is also important. For example you could open a connection per user session and don't close it until it expires.
2. Check your DB settings in order to find out if they are in par with your settings for connection pool.
3. Check if you have locking on a level of database. This was the reason which bogged down pretty tuned application I worked with before. The issue was with audit record where user accessing the application was logged. Transactions timed out all the time as each of them accessed this record for update

Answer 4

Hm 15 minutes sounds like a typical timeout duration, perhaps from a misconfigured DNS server . You should try to ping the hostnames of all involved systems.

Answer 5

Just read through an article on similar topic. Java SecureRandom entropy is about how SecureRandom implementantion caused application to lag for about 20 to 30 minutes. Maybe some debugging methods from it could help You to sort it out.