Retrieving all objects in code upfront for performance reasons

Question

How do you folks retrieve all objects in code upfront?

I figure you can increase performance if you bundle all the model calls together?

This makes for a bigger deal, especially if your DB cannot keep everything in memory

def hitDBSeperately {

get X users ...code

get Y users... code

get Z users... code

}

Versus:

def hitDBInSingleCall {

get X+Y+Z users

code for X code for Y...

}

Answer 1

Are you looking for an explanation between the approach where you load in all users at once:

# Loads all users into memory simultaneously
@users = User.all

@users.each do |user|
  # ...
end

Where you could load them individually for a smaller memory footprint?

@user_ids = User.connection.select_values("SELECT id FROM users")

@user_ids.each do |user_id|
  user = User.find(user_id)

  # ...
end

The second approach would be slower since it requires N+1 queries for N users, where the first loads them all with 1 query. However, you need to have sufficient memory for creating model instances for each and every User record at the same time. This is not a function of "DB memory", but of application memory.

For any application with a non-trivial number of users, you should use an approach where you load users either individually or in groups. You can do this using:

@user_ids.in_groups_of(10) do |user_ids|
  User.find_all_by_id(user_ids).each do |user|
    # ...
  end
end

By tuning to use an appropriate grouping factor, you can balance between memory usage and performance.

Answer 2

Can you give a snipet of actual ruby on rails code, our pseudo code is a little confusing?

You can avoid the n+1 problem by using eager loading. You can accomplish this by using :includes => tag in your model.find method.

http://api.rubyonrails.org/classes/ActiveRecord/Associations/ClassMethods.html