Rails threading - multiple tasks

Question

Hi

I am trying to run multiple tasks, each task access the database, and I am trying to run the tasks into separate execution wires.

I played around, tried allow_concurrency which I have set to true, or config.thread_safe! but it I get un-deterministic errors, for example sometimes a class is missing, or a constant ...

here is some code

grabbers = get_grabber_name_list
threads = []
grabbers.each { |grabber|
  threads << Thread.new {
    ARGV[0] = grabber
    if (@@last_run_timestamp[grabber.to_sym].blank? || (@@last_run_timestamp[grabber.to_sym] >= AbstractGrabber.aff_net_accounts(grabber, "grab_interval").seconds.ago))
    Rake::Task["aff_net:import:" + grabber].execute
    @@last_run_timestamp.merge!({grabber.to_sym => Time.now})
  end
  }
}
threads.each {|t| t.join }

thanks

Answer 1

You should probably do this using background workers. There are a few options for background worker libraries, but my favourite is delayed_job (http://github.com/tobi/delayed%5Fjob).

It should be pretty easy to convert the code you posted into background jobs.

Answer 2

I've recently implemented a Rails application that uses threads and made a few discoveries:

First, if you're writing to any arrays or hashes (i.e., complex types) outside your thread, wrap them in a mutex. It looks to me like hash and array references may not be thread safe. It seems unlikely that hash/array element indexing isn't thread safe but all I know is that after I put the external data structures in a mutex before writing, problems disappeared.

Second, close your ActiveRecord connection when the thread terminates, otherwise you can end up creating a large number of stale connections. Here's a post about how to do this. I don't know if it still applies for Rails versions > 2.2 but after I started closing connections explicitly, my problems disappeared. The author suggests monkey-patching ActiveRecord to do this automatically but I decided to release connections explicitly in my code.

Here's a sample of code that's working for me:

mutex = Mutex.new
my_array = []
threads = []
1.upto(10) do |i|
  threads << Thread.new {
     begin
       do_some_stuff
       mutex.synchronize {
         # You'd think that each thread would only touch its own personal
         # array element but without a mutex, I run into problems.
         my_array[i] = some_computed_value
       }
     ensure
       ActiveRecord::Base.connection_pool.release_connection
     end
   }
}
threads.each {|t| t.join}

By the way, if you're using threads to take advantage of multi-core CPUs, you'll need to use JRuby. As far as I know, JRuby is the only implementation that can take advantage of native CPU threads. If you use threads so you can do other things while waiting on network connections or some other non-CPU task, this isn't an issue.