Best way to create REST API for long lasting tasks?

Question

Suppose I have 2 servers.

The first is a service that provides some computations, which can last long time (minutes to hours).

The second server will use this service to have some data computed.

I'm trying to design a REST API for the first server and so far so good. But I'd like to hear some opinion on how to model notifications when the long lasting task is finished.

I considered 2 approaches so far: 1) Polling - the second server will ask every now and then about the result. 2) Callback - Second server will setup an uri for the first one to call after it is done. But this smells in REST API.

What do you think?

Answer 1

If you use Python, you can take advantage of RabbitMQ and Celery to do the job. Celery lets you create an item in a queue and then pause execution of whatever you're running it through (i.e.: Django) such that you can consume the output of the queue processor as it becomes available. No need for polling OR callbacks.

Answer 2

In addition to what I've already answered in this similar question, I'd suggest using the Atom Publishing Protocol for the notification (you could publish to your second server).

Answer 3

For your situation I would choose polling. When the second server makes the initial request to create the job on the first server, it should get a response that has the url of the eventual status page. The second server then polls that url every 5-15 minutes to check the status of the job. If the first server makes that url an RSS or Atom feed, then users could also point their RSS readers at the same url and find out themselves if the job is done. It's a real win when both people and machines can get information out of a single source.