How to run some tasks in the background and put the results to a global memory hash?

Question

I'm building a website with Rails which can let user check if some domains have been registered. The logic is designed like this:

1. There is a text field on a page let users input some domain names
2. When user click check button, the input will be post to the server
3. server get the inputs, create a background task(which will be executed in another process), and return some text like "checking now..." to user immediately
4. The background task will post the domain names to another site, get the response, parse it to get useful information, then put data to a globle memory hash (the task takes 3 seconds)
5. The page contains "checking now..." has a javascript function, will request to server to get the check result. It runs every 2 seconds until it gets result.
6. There is a action in the server side to handle the "check-status request". It checks that Globle memory hash, it found the result, returns it, otherwise return "[]"

(I'm using Rails 2.3.8, delayed_job)

I was a Java developer, this is what I will do in Java. But I found it difficult to hold a "globle memory hash" for "delayed job worker" and "rails", because they are in different processes.

Is my design is impossible in rails? Must I store the checking result to database or something like "memcached"?

And, can these background tasks run in parallel?

Answer 1

It's not just between DelayedJob's workers and Rails' workers that you won't see the global variables, in production you will be dealing with multiple Rails worker processes as well. So even if you were to set a global variable in Rails, some other Rails worker won't see it.

This is by design. It has the advantage that you can spread the load of Rails and DelayedJob across multiple machines, because they only deal with the stateless request, and look at the database system or other persistent storage to add the statefulness that your web application needs.

From what I've gathered, a Java web application may use a threaded model that would allow you to perform background tasks and set global variables like you want to. But that also limits you to a single machine; what would you do if you had to scale up?

Memcached actually sounds like a really good solution in this case. It's a snap to install, requires very little set up, and is easy to use from within Rails too.