views:

276

answers:

3

Hey,

At work we found that on some instances (particulary the slow ones) we have a different behaviour, acquired at the reboot.

We guess a cache is not initialized correctly, or maybe a concurrency problem... Anyway it's not reproductible in any other env than production.

We actually don't have loggers to activate... it's an old component...

Thus i'd like to know if there are tools that can help us to see the different objets present in the JVM memory in order to check the content of the cache...

Thank you!

Edit:

I don't have access to the production servers directly, our app server is weblogic 10, i don't have a pointer to the object but i know the cache object type...

Edit2:

Our servers are running on jre 1.5, is it possible to use jmap? Can't find it in a jdk5 :( Also, remote debugging could be nice but we can't for security reasons...

Edit3:

Actually jhat + VisualVM is ok for me, i found my object in dump but i'm not able to read the hashmap (object containing about 60000 items) properly... Is there a tool to read a concurrenthashmap in a friendly way? i need to find the value of a key (or its existence in the map) without browsing manually the 60k records. Actually i read on eclipse MAT forum that it's also not possible with it...

+2  A: 

Do you have access to a handle/pointer to the objects? If so, you can fire it up in debug mode, and look at it in a debugger like Eclipse. This will allow you to inspect variables and the like.

Alternatively, you could write a little logger that reflectively walks through the class and logs what is going on. All of this assumes you have a location where you can start to step into the code, or can get a reference to the cached values.


edit: As I noted in my comment, I do not know of a way to get access to an object without some sort of reference or link of references. For instance, is the object encapsulated in another object? If so, you can do something like the following:

Class<?> objectClass = myPointer.getClass();
Field[] objectFields = objectClass.getDeclaredFields();
for (Field field : objectFields) {
    field.setAccessible(true);
    //Or whatever you would need to do to get the information you need
    System.out.println(field.get(myPointer).toString());
}

You can also do something like:

Field targetField = objectClass.getDeclaredField("myFieldName");
targetField.setAccessible(true);
MyOldObjectType target = (MyOldObjectType)targetField.get(myPointer);
//do whatever you need to do here

Note that all of the reflection methods throw exceptions, so you will need to handle those as appropriate. Also, setAccessible(true) means you can access private fields and methods on the object. This is EXTREMELY DANGEROUS - Only use it if you absolutely have to.

aperkins
I don't think a debugger like that present in Eclipse is an option because "it's not reproductible in any other env than production."
Andrew Hubbs
@Andrew Hubbs: I have personally used Eclipse debugging in a production environment - you must start up the system in debug mode, and set it up to remote connect. There are options there - it is not easy, but it is possible.
aperkins
our servers are not in the same city, i don't know if they can start a production weblogic in debug mode, and don't know either if they would give me access through a tunnel...
Sebastien Lorber
+1 Remote debugging would be the best thing to try next.
rob
@Sebastien Lorber: I would suggest using @Will Hartung's suggestion then - you could get a heapdump and then parse through it with the various tools suggested. Just as an FYI - the first time I saw a remote debug session, it was to servers in another city as well. It is possible - but it sounds like the heapdump would be a better option for you.
aperkins
it is yes for sure i actually debug on remote servers everyday, but they would never give me access to a production instance...
Sebastien Lorber
@Sebastian Lorber: I can understand that - the few times I have been involved in this, it has been very specialized instances, where we were unable to figure out what was wrong any other way, and everything was done through our internal IT group so they could sign off on it.
aperkins
+4  A: 

It's fairly easy to run jmap to dump a COUNT of object instances, but I don't know if that's what you're really interested in.

You can also use jmap to do a heap dump of the entire heap, and with that (and jhat), you can see object RELATIONSHIPs (i.e. what object points to what), but not necessarily object CONTENTS.

Certainly the data is there in the heap dump, it's not "click away" visible.

Some of the professional profilers I think will let you introspect objects in a heap dump.

Otherewise you'd be better off adding some specific instrumentation to the application to provide the specific introspection you're looking for, triggered by custom code, JMX, or whatever.

Will Hartung
+1 - I didn't know about the heap dump. That would have been really useful a month or two ago for me :)
aperkins
We use java 1.5Is it possible to use jmap on a 1.5 jre? I've found jmap only on jdk6 and it couldn't work with my local jre 1.5...
Sebastien Lorber
+3  A: 

This is basically to expand on what Will said. I've had great success by having our admins do a heapdump of our production systems, though with the caveat that the particular server you are dumping will be non-responsive until the dump completes. Then getting that file and using the Eclipse MAT plugin to look at it. If you don't like Eclipse, Netbeans and the plain VisualVM plugin can be used as well. This can create some big files though, you may need to run on a 64 bit system.

mezmo
+1 for VisualVM
ykaganovich
+1 for Eclipse MAT - imo the most powerful (free) heap visualizer out there. Highly recommended, and has awesome plugins to look at the heap in interesting ways (collection fill %, for example!).
Trent Gray-Donald