Java framework/tool for simple distributed computing problem

Question

Hi

We generate pdf files with data regarding monthly financial balance of tens of thousands of clients. At it's peak (100.000 files at the end of year), the process may take as long as 5 days to complete using distribute the load between 5 servers. The distribution of workload is a manual process (eg. server 1 generates pdf for clients 1 to 20.000, server 2 from 20.001 to 40.000, and so on).

We use Java, so we would like to use a Java tool or framework in a fashion similar to BOINC (BOINC is not coded in Java), which would allow us to have:

• a server automatically distributing data and code to workers, and managing them
• workers (plain dual core desktops using free (or not so) cpu cycles) receiving code sent by server and executing it using the data as input, giving some status feedback, and finally doing something with the output, eg. sending pdf generated files to network storage.

For now, we don't want to try heavyweight all-powerful toolkits such as Globus Toolkit.

¿Do you know any java framework/tools suitable to our needs? Open source would be nice, but we don't discard commercial solutions.

Answer 1

There are many:

• http://www.jppf.org/
• https://jxta.dev.java.net/
• https://cajo.dev.java.net/
• http://www.dmoz.org/Computers/Programming/Languages/Java/Class%5FLibraries/Networking%5Fand%5FDistributed%5FComputing/

Google is your best friend here.

Answer 2

• http://hadoop.apache.org/
• http://www.gridgain.com/
• http://www.oracle.com/technology/products/coherence/index.html

Answer 3

Google says there is a Java port of BOINC here: http://boincoid.sf.net

It's mostly targeting about android :-/ but apparently all but the GUI can run on a regular VM.

Answer 4

Another one you can look at is Terracotta.

Answer 5

Check out Hazelcast distributed executor service. It uses Java's standard ExecutorService API.