How can I query mongodb using mongoid/rails without timing out?

Question

I have a rake task that processes a set of records and saves it in another collection:

batch = [] 
Record.where(:type => 'a').each do |r| 
  batch <<  make_score(r) 
  if batch.size %100 == 0 
    Score.collection.insert(batch) 
     batch = [] 
   end 
end 

I'm processing about 100K records at a time. Unfortunately at 20 minutes, i get a "Query response returned CURSOR_NOT_FOUND" error. The mongodb faq[1] says to use skips and limits or turn off timeouts. I found the skips and limits was about ~2-3 times slower.

How can i turn off timeouts in conjunction with mongoid?

[1] http://www.mongodb.org/display/DOCS/Frequently+Asked+Questions+-+Ruby#FrequentlyAskedQuestions-Ruby-IkeepgettingCURSORNOTFOUNDexceptions.What%27shappening%3F

Answer 1

The MongoDB docs say you can pass in a timeout boolean, and it timeout is false, it will never timeout

collection.find({"type" => "a"}, {:timeout=>false})

In your case:

Record.collection.find({:type=>'a'}, :timeout => false).each ...

I also recommend you look into map-reduced with Mongo. It seems tailer made to this sort of collection array manipulation: http://www.mongodb.org/display/DOCS/MapReduce