views:

84

answers:

1

Hi

I have a production web application running in Tomcat. The web application uses struts 2 as MVC layer.

We had an issue wherein one of the web servers spiked to 100% cpu usage. This issue lasted for over several hours. I took the thread dump and saw over several hundred threads in runnable state and the dumps show the same stack trace for most of the threads.

TP-Processor2" daemon prio=10 tid=0x00002aab80880c00 nid=0x5b4f runnable [0x0000000043bff000..0x0000000043c05d90]
   java.lang.Thread.State: RUNNABLE
        at java.util.HashMap.get(HashMap.java:303)
        at com.opensymphony.xwork2.util.LocalizedTextUtil.buildMessageFormat(LocalizedTextUtil.java:620)
        at com.opensymphony.xwork2.util.LocalizedTextUtil.getDefaultMessage(LocalizedTextUtil.java:588)
        at com.opensymphony.xwork2.util.LocalizedTextUtil.findText(LocalizedTextUtil.java:461)
        at com.opensymphony.xwork2.TextProviderSupport.getText(TextProviderSupport.java:224)
        at com.opensymphony.xwork2.ActionSupport.getText(ActionSupport.java:99)
        at org.apache.struts2.components.Text.end(Text.java:158)
        at org.apache.struts2.views.jsp.ComponentTagSupport.doEndTag(ComponentTagSupport.java:43)
        at org.apache.jsp.parts.myjsp_jsp._jspx_meth_s_005ftext_005f2(myjsp_jsp.java:296)
        at org.apache.jsp.parts.myjsp_jsp._jspService(myjsp_jsp.java:94)

Now the code in question uses struts s:text tag and just fetches the value from a property file. I am not sure why so many threads should get stuck over there (threads are in runnable state).

Can i get some help in resolving what could have gone wrong.

+3  A: 

A guess right out of left-field: you're not synchronizing multithreaded access properly.

java.util.HashMap is not threadsafe, and if accessed concurrently by multiple threads, one particular mode of failure is an infinite loop within a particular bucket.

Take a look at the map in question (indicated by the stacktrace), and see whether multiple threads are in fact able to access it without adequate synchronization.

Andrzej Doyle
That sounds right to my ears. I think a concurrent Hashmap exists off the bat. Or you might be accessing the same file multiple times and lock yourself out.
Kdansky
@Kdansky - yep, `java.util.concurrent.ConcurrentHashMap`. But the map exists within a third-party class, so chances are this might be down to sharing some higher-level component incorrectly rather than choosing the wrong map implementation.
Andrzej Doyle
I have a doubt here. Even if this means synchronization issue, the threads should have been waiting on something. Why are there so many threads in runnable state. Also, all these threads are in this state only when trying to retrieve a particular key from that map. I do not see any thread in the dump which is in runnable state when trying to fetch any other key.
Sushman
If the code eventually only fetches localized texts from the HashMap, it sounds unlikely that the map is modified at the same time as these threads are accessing it with a read operation. Even if it's not guaranteed by the API spec, the current implementation of java.util.HashMap#get(key) is thread safe.
jarnbjo
@Sushman - that sounds **exactly** like the problem I described. It sounds like you've got some preconceptions about what a thread-safety failure "looks like" - if you read the link I provided this will hopefully clear this up. The failure I'm talking about puts the HashMap in a state where threads traversing beyond a particular point in a given bucket (i.e. for a small subset of keys) go into an infinite loop (i.e. are in RUNNABLE state at 100% CPU). That's ironically exactly what you've just described in your reasons to *dis*believe it. :-)
Andrzej Doyle
@Sushman this is the right answer. Don't add to a HashMap while other threads could be in it.
bwawok
Thanks for your explanations. The map should have been populated initially at application startup after which only get operations are permissible in the map. But since it belongs to a third party class (Struts framework), I do not have a clear idea on how this was implemented. I will take a look at struts source code and get back on this.
Sushman