ansaurus

Question

Java thread blocking

Answer 1

A:

You should replace your logging library with Logback. It is thread safe and will most probably prevent this kind of problems in the future.

There are number of question tags with the [logback] tag here on SO.

Boris Pavlović 2010-08-20 14:25:18

Unfortunalty i can't change the logging library.

Alexandre Boudreault 2010-08-20 15:31:40

Replacing non thread safe logging library with Logback will solve the problem. Everything else is just patching without warranty. In this situation you can escalate the problem, suggest the solution and hope for the best.

Boris Pavlović 2010-08-20 15:55:08

Answer 2

A:

I never used java.util.logging, so I don't know whether my suggestion is useful, but netherless:
try to use different instance of java.util.logging.Logger, so not all 240 threads will be blocked on the same monitor
(it will help if different instances of Logger use different instances of java.util.logging.ConsoleHandler) .

Victor Sorokin 2010-08-20 14:26:51

Answer 3

A:

It seems that the thread that own "0x00007fe37e72b340" is blocked at the IO level. Maybe a disk (raid?) issue?

can you do a thread dump 5 minutes later the see if the same thread is still blocked?

Thierry-Dimitri Roy 2010-08-20 14:27:12

Thanks for the tips, ill try to monitor thread activity during the next crash in a couple of hour ;)

Alexandre Boudreault 2010-08-20 15:32:47

Answer 4

+3 A:

All your Threads are logging things. They all need to write on the disk from time to time. Everytime one of your 240 threads hits a logging line there will be disk access issues.

It baffles me that the Thread having the lock is in the RUNNABLE state.

I think it might be waiting for some external ressource to be released (like disk access for example)

Are you running low on disk space? Have you recently changed something in your storage system?

BenoitParis 2010-08-20 14:29:56

I think this is the right track. I'd look at external factors. Also, if it's writing not to a traditional file but to a Unix named pipe, make sure that somebody is reading the other end of the pipe at a sufficient pace. If the buffer fills up you'll just block.

Mark Peters 2010-08-20 14:34:22

The disk space is fine and we did not changed anything on the system. We tried to switch the system on another server and we got the same prob.

Alexandre Boudreault 2010-08-20 15:30:50

I don't think this is an issue with Thread safety. It would have broken much earlier.240 Threads is not a problem for the JVM.240 Threads editing a single file could be.An ugly fix would be to reduce the amount of logged messages.Try tweaking the thing using getLoggerNames(), getLogger() and setLevel() fromhttp://download-llnw.oracle.com/javase/1.4.2/docs/api/java/util/logging/LogManager.html andhttp://download-llnw.oracle.com/javase/1.4.2/docs/api/java/util/logging/Logger.html

BenoitParis 2010-08-20 16:00:58

Answer 5

A:

Flushing after each log record is going to expensive if you have very verbose logs.

A quality fix would be to clean up the logging, probably based around auditing.

As a quick fix, override StreamHandler.flush or OutputStream.flush to not do so immediately. Only flush once every so often. Note, however that you could potentially lose logging data immediately before a crash if you do this.

Tom Hawtin - tackline 2010-08-20 14:35:08

ansaurus

tags:

views:

answers:

Java thread blocking

related questions