Running long experiments from a web interface.

Question

Hi all, I'm building a research web app that runs a long batch analysis that can last hours. I'm using Grails but I think this is a general design issue. The batch process can be started by a URL, e.g.

http://mydomain/experiment/run?a_lot_of_params

And this page returns info about the experiment (whose data are stored in the back end and are accessible through another web interface). e.g.

Experiment finished: results available at 
http://mydomain/experiment/data/myexperiment.xml

The problem with this approach is: if the experiment lasts a few minutes it's fine to leave a web page waiting. But what happens if the experiment lasts 10 hours? What if the user closes the page? Is there a correct design pattern to follow for this type of situation?

Thanks for any hints.

Answer 1

You can give user a token and make them enter that token to view the status of the process.

At the end of the process you can notify user via email/call/sms .

Answer 2

Is the batch process asynchronous? If not, you'll need to make it run in another thread. An example:

class BatchController {

    def batchExecutionService // has the logic to run the batch operation

    // this should probably only take POST requests, but that's left as a later exercise
    def execute = {

        def batchId

        new Thread(new Runnable() {
            batchExecutionService.executeBatch(batchId, params)
        }).start()

        render batchId // could also set response status to 202 Accepted
    }

    def check = {
        render batchExecutionService.getStatus(params.batchId)
    }

    def retrieve = {
        if(batchExecutionService.getStatus() != Status.FINISHED) {
            // error response, could be 404 if report is not created yet, or something else
        }
        render batchExecutionService.getReport(params.batchId)
    }
}

The above implementation is pretty crude, but conveys the general message. A couple notes on it:

• You probably wouldn't want to just new-up a Thread and use it, there are better ways to do it (but would yield a more complicated example). A possible solution would be the BackgroundThread plugin or Quartz plugin.
• Arguably, the execute action shouldn't take GET requests with query parameters like you're currently doing. But that's an entirely different discussion. See this for a reference.