Searching for Generic Asynchronous Java Job Execution Framework / Library

Question

I am looking for a generic asynchronous Java job execution framework that could handle Callables or Runnables. It would be similar to java.util.concurrent.ExecutorService, (and possibly wrap ExecutorService), but it would also have the following features:

1. The ability to persist jobs to a database in case the application goes down while a job is being serviced, and be able to restart the unfinished jobs. (I understand that my job may have to implement Serializable which is OK.)

2. Work with UUIDs to enable the client to obtain job tokens and inquire about job status. (Under the hood this information would be persisted to a database, as well.)

I have started working on this myself by building around ExecutorService, but I would prefer an out of the box, open source solution, if one exists.

Something that could work within the Spring Framework would be ideal.

Answer 1

You may want to look at Quartz.

Quartz is a full-featured, open source job scheduling system that can be integrated with, or used along side virtually any J2EE or J2SE application - from the smallest stand-alone application to the largest e-commerce system. Quartz can be used to create simple or complex schedules for executing tens, hundreds, or even tens-of-thousands of jobs; jobs whose tasks are defined as standard Java components or EJBs. The Quartz Scheduler includes many enterprise-class features, such as JTA transactions and clustering.

Answer 2

You can use Quartz, and create a concrete Job adapter that delegates to a Runnable or Callable. Quartz' Job interface adds the ability to maintain some state between invocations of a task. If desired, Quartz can store jobs and their state durably in a relational database, and execute them on a scalable cluster of hosts.

Answer 3

Take a look to http://www.opensymphony.com/quartz/wikidocs/Features.html and see if it has already something for you.

From that page:

With the use of the included JDBCJobStore, all Jobs and Triggers configured as "non-volatile" are stored in a relational database via JDBC

Answer 4

Another direction might be something like using Terracotta, which has the ability to cluster heap in your JVM and persist it for availability. Terracotta supports integration with Quartz if that's useful from a scheduling point of view. Also, there is a master-worker and messaging integration module that might be useful as well. Terracotta is open source.

Answer 5

To follow up on Alex's point, a Terracotta solution wouldn't persist your jobs to the Database, they would be persistent in the Terracotta distributed memory store.

Since Terracotta persists the memory store to disk, this is a more efficient version of putting those jobs into the database.

At the same time, it gives you a pure POJO programming model, so you don't even have to deal with DB txns, ORM and the like - unless your particular workload happens to talk to the DB (in which case Terracotta doesn't help or hurt you here, it just helps distribute the work).

The MasterWorker pattern will help you distribute work out on the grid, and you can very easily get started using a DistributedExecutorService, submitting work looks like this:

CompletionService executor = new DistributedCompletionService(new DistributedExecutorService("myTopologyName"));
executor.submit(new MyRunnable(), null);
...
Future f = executor.take();

Here's the link to Quickstart guide in the master-worker implementation on the Terracotta Forge.

What's more - Terracotta doesn't require that you implement Serializable - although you can if you want to :)