Long Application Startup times on WebSphere Application Server 6.1 FixPack 21

Question

Hi,

I'm seeing that my application startup times have grown long after I've migrated from WAS FixPack15 to FixPack 21. By application startup time, I mean the interval between the log statements :

WSVR0200I: Starting application

AND

WSVR0221I: Application started

This interval was less than 10 sec on FP 15, but it is 2 min on FP 21. Nothing has changed, except for the new fix pack. However, if I bounce my WAS instance, and then start the application, the start up time was short (10 sec). But on occasions when I don't bounce my WAS instance, my application startup on FP 21 takes 2 min where it used to take 10 sec on FP 15.

I was wondering if any of you have any pointers about how to troubleshoot this issue.

My OS is Red Hat Enterprise Linux Server release 5.1 (Tikanga)

Answer 1

One thing that I would do is time it by comparing the logs against the faster version and the slower version. This may allow you to narrow the poor performance down to an individual component rather than the entire thing.

I have used 21 in the past and didn't notice it being slower, but that said I was using it on AIX and Windows so it may be an operating system issue.

If you narrow down the poorly performing function post it here and then I will look at it a bit more.

Regards,

Michael.

Answer 2

I went from FP13 to FP21 and didn't see any slowdown on my apps, but that was on Windows.

Are you sure that's the only thing you have changed?

Are you using any WebSphere APIs / code in your app?

Did you actually apply all of the fixpacks? The download page is a real eyechart, but you have to update WebSphere and the Java SDK. If you are using IHS, you have to update IHS, the plugins and the SDK there too.

Finally, you say it's only slow when restarting the app, but not when restarting the whole server. I am wondering if there's JVM heap issues with your app. How big of a heap are you running with? Are you seeing GC issues when restarting it?

Answer 3

A few ideas:

1. Take a few javacores (kill -3 PID), 30 seconds apart, during the 2 minute pause and use something like Thread and Monitor Dump analyzer to compare the multiple thread dumps to see if a thread is "stuck" somewhere, making an external call or looping. See the Linux Hang MustGather: http://www-01.ibm.com/support/docview.wss?uid=swg21115785
2. Check verbosegc during that 2 minute interval (with something like Garbage Collection and Memory Visualizer) to see if there's a GC pause at that time (a 2 minute pause would mean you have other problems).
3. Try to reproduce the problem in a test environment. There you could do heavier debugging (such as #4).
4. Dynamically turn on *=all right before starting the application. Obviously this will be a huge performance hit, but unless this is a timing issue, you should be able to see any "gaps" in timestamps or superfluous activity of some sort. Use Trace Analyzer for WebSphere Application Server to manage the trace.log files.