Multi-node concurrency in Java

Question

I've written a multi-threaded Java program to solve an embarrassingly parallel problem such that it utilizes all the free CPU cycles of on a multi-core CPU. I'd like to refactor my solution so that it can run on multiple nodes while still keeping the majority of the code I've already written.

I've used MPI with C in the past and been told that it's the "correct" way to address the issue of maximizing CPU cycles, but I'm also aware of other concurrent frameworks in Java like RMI and wonder if they are just as good.

Is there a good way to handle multi-node and multi-core concurrency in Java where the main goal is to leverage the most CPU cycles as possible out of the cluster?

Edit: I get the impression that there's no easy way to handle this stuff. I'm not surprised, but I was hoping. :)

Answer 1

JMS is a good place to start.

Answer 2

You can try Hazelcast. It has a distributed ExecutorService. This should allow you to add tasks to a service which run across any number of nodes.

Answer 3

Depends on what you are doing and your budget you might want to look into (in no particular order)

• Actors especially Akka that has good remote actors, STM and supervisor style managment with a Java API
• Norbert
• GridGain
• Terracotta
• Gigaspaces
• Oracle Coherence
• IBM Extreme Scale
• TIBCO ActiveSpaces

Also for non distributed systems

• java util concurrent and guava
• Libraries such as javolution or JSR 166
• Functional programming capable JVM languages such as Scala and Clojure has better multi core utilization than Java.

Answer 4

Also consider Apache Hadoop, it uses MapReduce and is well suited for many parallel solutions.