Ruby Mongo or Mongoid concurrency problem

Question

I am having what I think must be a concurrency problem. I am using passenger, rails 2.3.5, mongoid 1.9.2, and mongo ruby driver 1.0.9. I am using jQuery to request data that is pulled from MongoDB and then rendered in the browser. Everything is working great until I started to make two such requests at the same time. In the model these are the methods that get executed by the requests:

Class Visit
  include Mongoid::Document
  ...
  def self.cancellations_and_visits_by_therapist_graph(clinic_id) 
    visits = collection.group("function(x){ return { resource_id : x.resource_id } }",
            {:clinic_id => clinic_id, :visit_date => { "$gte" => Time.now - 6.months, "$lte" => Time.now}}, 
            {:visits => 0, :cancel_no_shows => 0},
            'function(obj, count) {
              if (obj.visit_status == "NO SHOW" || obj.visits_status == "CANCELLED") {
                count.cancel_no_shows += 1;
              } else {
                count.visits += 1;
              }

            }')

    visits = visits.group_by {|g| g['resource_id']}

    Resource.any_in(:mysql_id => visits.keys).order_by([:last_name, :asc]).order_by([:first_name, :asc]).inject({ 'visits' => [], 'cancel_no_shows' => [], 'xlabels' => []}) do |formatted_visits, resource|
      formatted_visits['visits'] << visits[resource.mysql_id.to_f].first['visits']
      formatted_visits['cancel_no_shows'] << visits[resource.mysql_id.to_f].first['cancel_no_shows']
      formatted_visits['xlabels'] << resource.last_name + ", " + resource.first_name
      formatted_visits
    end
  end



  def self.total_visits_for_graph(practice_id)
    visits = collection.group("function(x) { return { clinic_id : x.clinic_id } }", 
                          {:practice_id => practice_id, :visit_status => 'COMPLETE', :visit_date => { "$gte" => Time.now - 6.months, "$lte" => Time.now}},
                          {:visits => 0}, "function(obj, count) { count.visits += 1; }")

    visits = visits.group_by {|g| g['clinic_id']}
    Clinic.any_in(:mysql_id => visits.keys).order_by([:name, :asc]).inject({ 'data' => [], 'xlabels' => []}) do |formatted_visits, clinic|
      formatted_visits['data'] << visits[clinic.mysql_id.to_f].first['visits']
      formatted_visits['xlabels'] << clinic.name
      formatted_visits
    end
  end
end

The best way to describe the problem is that results from Mongo are getting passed to the wrong object. I lookated an example of this:

This was returned when I called CLinic.any_in (it is a result from one of the groups):

{"group"=>{"$keyf"=>"function(x){ return { resource_id : x.resource_id } }", "cond"=>{:clinic_id=>101, :visit_date=>{"$gte"=>Tue Apr 20 15:34:37 +0800 2010, "$lte"=>Wed Oct 20 15:34:37 +0800 2010}}, "ns"=>"visits", "initial"=>{:visits=>0, :cancel_no_shows=>0}, "$reduce"=>"function(obj, count) {\n              if (obj.visit_status == \"NO SHOW\" || obj.visits_status == \"CANCELLED\") {\n                count.cancel_no_shows += 1;\n              } else {\n                count.visits += 1;\n              }\n\n            }"}}

This (a clinic object) was returned by the collection.group call:

{"_id"=>BSON::ObjectId('4cb7d72b3bc5457800ce2e6f'), "name"=>"Corona", "practice_id"=>39, "mysql_id"=>101}

As with all good concurancy problems the results are randome, sometimes it works fine and sometimes it blows up. I am new to mongo and mongoid so I am actually not sure if this a problem with mongoid or the mongo driver, however I think it is related to Mongoid. I am including my initializer that I am using to load Mongoid in rails. Any ideas or even just additional debugging ideas are greatly appreciate.

Connection

mongoid_conf = YAML::load_file(Rails.root.join('config/mongoid.yml'))[Rails.env]

Mongoid.configure do |config|
  config.master = Mongo::Connection.new(mongoid_conf['host'], 27017, :pool_size => 5, :timeout => 5).db(mongoid_conf['database'])
end

Answer 1

I found the solution to this problem. It was actually not either the Mongo driver or Mongoid, it was passenger. When Passenger spans the rails process it forks the current instance so the file descriptors (including the TCP descriptors get shared across app instances). This means that that mongo is being written and read to by the same socket, which is what lead to the concurrency problem. The solution is to make mongo reconnect whenever it forks. This is the solution that I found:

# Handle the creation of new processes by Phusion Passenger
if defined?(PhusionPassenger)
  PhusionPassenger.on_event(:starting_worker_process) do |forked|
    if forked
      # We're in smart spawning mode.

      # Reset the connection to MongoDB
      Mongoid.config.master.connection.close
      load File.join(RAILS_ROOT, 'config/initializers/mongoid_init.rb')
    else
      # We're in conservative spawning mode. We don't need to do anything.    
    end
  end
end

Reference http://groups.google.com/group/mongodb-user/browse_thread/thread/f31e2d23de38136a for the original answer posting and discussion.