Java distributed objects with locality?

Question

I am evaluating various Java object distribution libraries (Terracotta, JCS, JBoss, Hazelcast ...) for an application server and I'm having trouble understanding their behavior on various axes.

My requirements for distributed objects are not many -- they boil down to one-to-one and one-to-many messaging. There's more, but for the rest we just use JDBC and I assume I can plop a cache in front of this using any of the available libraries.

I would like a system that distributes objects and exhibits locality properties -- in other words, a server that grabs an object tends to hold onto it without excess communication to other nodes. Hazelcast looks simple (and peer-to-peer is nice) but seems to require objects are distributed evenly across all nodes.

I'd like a way to persist objects, preferably transparently. I plan on using EC2, so I have the option of temporary, free, limited local storage (the disk) and permanent, non-free, unlimited storage (S3). It'd be great not to worry about OutOfMemoryErrors.

I like the simplicity and "magic" of Terracotta but it scares the beejeezus out of me. Also in order to truly scale you have to spend $$$$, otherwise you're communicating with a single hub.

I'm cheap and I want something not only free but mature and with a large userbase.

Thanks for any input.

Answer 1

Terracotta seems like a perfect fit for your situation.

• It's simple to setup
• it can be configured to be persistent (use an EBS volume for EC2)
• it's closely integrated with Ehcache (actually Terracotta bought Ehcache) for great distributed caching performance
• the free offering scales pretty well with several clients.

Just start playing around with it. I bet you'll love it. To ease your performance fears, simply run a through put test for message passing. This shouldn't take much more than an afternoon of your time.

I have to admit that I haven't used Terracotta for a year and that I don't know the others you suggested.

Answer 2

Terracotta does fit the bill. I understand your objections, but here's my comments:

1) Terracotta does exhibit locality - and is probably the best system at it compared to those you mentioned. Objects are only brought in to a local JVM where requested. Locking for reads or writes is performed using a leasing mechanism. This means if you exhibit perfect locality in your system then you will incur very little network overhead.

2) Terracotta provides disk persistence out of the box - in the OSS version (you don't have to pay $$$$)

3) Why does it scare you so much? Just use EHCache as a cache, or the Hibernate 2nd Level Plugin. It's incredibly easy to setup and use.

4) Yes, Terracotta FX requires you to pay (for scale-out servers). However I would suggest that if you have a system that is mostly read and exhibits true locality then I don't think you'll have a problem getting the scale you are looking for. With Terracotta 3.2 the performance of the Hibernate 2nd Level Cache is 100,000 ops/s using 8 application servers and one Terracotta server at 100/0 read/write ratio and 12,000 ops/s using the same config at 95/5 read/write ratio.

(I just did a talk for the Bay Area SDForum on these numbers so I happen to have them handy)

Answer 3

Btw, it's not clear what you are looking for (messaging is not the same as clustering/distributed objects).

If you are looking for messaging in Java I recommend you have a look at RabbitMQ (it's Erlang based but that doesn't matter).

Answer 4

Hi,

Yes Hazelcast will distribute your objects across the cluster. However you can enable near cache if you want to reduce the communication cost. http://www.hazelcast.com/documentation.jsp#MapNearCache