Peculiar JRE behaviour running RMI server under load, should I worry?

Question

I've been developing a minimalistic Java rich client CRUD application framework for the past few years, mostly as a hobby but also actively using it to write applications for my current employer.

The framework provides database access to clients either via a local JDBC based connection or a lightweight RMI server. Last night I started a load testing application, which ran 100 headless clients, bombarding the server with requests, each client waiting only 1 - 2 seconds between running simple use cases, consisting of selecting records along with associated detail records from a simple e-store database (Chinook).

This morning when I looked at the telemetry results from the server profiling session I noticed something which to me seemed strange (and made me keep the setup running for the remainder of the day), I don't really know what conclusions to draw from it.

Here are the results:

Memory
GC activity
Threads
CPU load

Interesting, right?

So the question is, is this normal or erratic? Is this simply the JRE (1.6.0_03 on Windows XP) doing it's thing (perhaps related to the JRE configuration) or is my framework design somehow causing this?

Running the server against MySQL as opposed to an embedded H2 database does not affect the pattern.

I am leaving out the details of my server design, but I'll be happy to elaborate if this behaviour is deemed erratic.

Answer 1

Short answer: no, I don't think this looks scary.

I don't think you have enough information here to determine exactly where the spiky behavior comes from. There's nothing that indicates that there's a memory leak, thread leak, resource leak or obvious contention. Perhaps some of your tasks managed to get in step with each other? In any case, you seem to be observing correct behavior and nothing in the profile indicates dangerous symptoms.

That said, I would stronly recommend that you upgrade to the latest version of Java. Update 3 is too old: we're not even allowed to use it at work due to the security issues. There has been plenty of work done on the garbage collector since then as well.

As a first step, I would recommend upgrading to latest Java (update 20 as of this writing) and re-run your test. It's entirely possible that this puzzling behavior will vanish on the second try.

EDIT: with respect to the instant deadlock found in the comment, I wouldn't use that finding as an indication of anything other than multithreaded programming is hard (do-able but hard). I've recommended Java Concurrency in Practice before and I strongly recommend that you keep it nearby while coding.

That said, I would also recommend avoiding RMI if at all possible. The hard-wired coupling that creates between client and server (client hangs until server fulfills request) can cause another layer of distributed computing complexity that really isn't worth it for a simple "request + fulfill" pairing.

Congrats on finding the deadlock, though. There are plenty of times when I wish my own (RMI-caused) issues were as straightforward....