how to do background computing in Spring 3.0?

Question

In a spring application I want do some background computing. The task is that I save an entity( lets say a GPS coordinate) and then I want to query a web service to get distance between all the coordinates and store them in the db for later use. There is a possibility that after storing the entity, connection cannot be made to the web service which will be used for calculation of the distances.

I think that this can be accomplished with introducing a job in the system (some thing like a cron job). Once we want some thing to happen in the background we put it in a job queue. The queue will try to execute the first job it receives, A job will go and connect to the web service, if for some reason it fails, it stays in the queue and tells the queue that it failed. The queue can then execute after a specified time for retry.. When it's successfully a job complete it is taken out of the job queue.

I am clear on the requirements, but I am not sure which technology to use. JMX, JMS, Spring Task Scheduler, Spring Integration, mix/match or all.

This seems similar to http://stackoverflow.com/questions/1939229/how-to-create-background-process-in-spring-webapp but not entirely

Edit Let's add to it a bit further. There is also a possibility of additions of those GPS coordinates in bulk (via a CSV file) for each CSV file every entry will be processed in file (the order will be for first come first serve file and for each entry). Deletions case can be simpler as nothing is expected to be deleted (but can be deatl with later)

Answer 1

I'd use JMS and Spring message-driven POJOs. Use a guaranteed deliver queue to ensure that the message is put back on the queue in the event of a failure. Supply an error queue to handle max retry problems.

Answer 2

If you "only" need asynchronous execution, then the JMS approach suggested by duffymo is definitely a good solution (message delivery is guaranteed, it's fault-tolerant, transactional, etc). However, in case of failure, messages will be rolled back and immediately delivered, unless you configure a Redelivery Delay and a Redelivery limit. But while some JMS providers do offer such features (e.g. MQ, WebLogic, JBoss), this is not a standard thing AFAIK (for example, GlassFish OpenMQ doesn't offer this). You need to take this into account.

Another approach would be to use a scheduler (especially if you also want to delay or schedule the jobs execution) - and I'm thinking to Quartz here - and to reschedule jobs on failure (this is fully supported by Quartz). And Spring also provides nice integration of Quartz.

Answer 3

Of the technologies you mentioned, JMX isn't relevant to the problem, and while JMS would probably work, it is likely to be overkill (the amount of effort in setting it up can be considerable, depending on your platform). Quartz is also an option, although Quartz's strengths lie in cron-type scheduling and clustered scheduling, neither of which seems to be a requirement for you.

Your requirement seems to be to submit a task to a queue, execute those queued tasks one at a time, and if one of them fails for a specific reason, then hold up further processing for a certain amount of time, and then try again with the previous task.

The simplest approach I can think of is pretty similar to the other question you mentioned, which is to use the Spring TaskExecutor abstraction, but with a twist.

If your requirement is just that the queue should be "held up" for a fixed period of time if a web service failure is detected, then you can handle that entirely within the executing task, e.g.

public class MyTask implements Runnable {
   public void run() {
      while (true) {
         try {
            performWork();
            return;
         } catch (WebServiceFailure f) {
            sleepForFixedPeriod();
         }
      } 
   }
}

The configure a ThreadPoolTaskExecutor is Spring, with a maxPoolSize set to 1. Your client code then submits worker task objects (like the one above) to the executor.

If you were to increase the corePoolSize of the executor to a value higher than 1, then you could take advantage of multiple cores to improve throughput by allowing the executor use multiple threads, but the "catch-and-retry" logic would only apply on a per-thread basis.

It's not the most elegant of solutions, but it's very simple, easy to understand and setup.