One big object or one small plus one big info object?

Question

Hi,

For a Ruby on Rails app, I'm having this Movie object which has several attributes. Pretty much every page needs to compare some movies to a full list of these movies sorted by ratings, so I used to cache this big sorted list. It was useful, because I could directly use this list of Movies and all their attributes without any other DB request.

Recently, the list got bigger and a big array of 2500 movies just can't fit into a 1MB Memcache chunk.

I was wondering if splitting this Movie object into a smaller object (id, created_at, updated_at, score, attributes_id) and externalizing its attributes into another object was a good idea. I would have a smaller list of movies sorted by score, but lot of other requests.

While we're at it, why not even cache an array of 2500 integers, do the maths with Ruby and only get the 50 movies we're interested in? Would it require 50 DB calls, or can I ask PostGreSQL to give me an array of the movies 2, 5, 6, 89, …, and 2467?

What should I do? What other part of the problem am I missing?

Thanks you,

Kevin

Answer 1

Instead of stripping attributes out of your model, why not load in only what you need? It's often to your advantage to use the data directly instead of involving the models when doing simple numerical comparisons. The storage requirements for a simple hash are dramatically lower than a full ActiveRecord instance.

In this case, select_all is your friend:

class Movie < ActiveRecord::Base
  def self.data_for(*ids)
    select_all(
      sanitize_sql([ "SELECT id, score FROM #{quote_table_name} WHERE id IN (?)", ids.flatten ])
    ).inject({ }) do |h, row|
      h[row[:id].to_i] = row
      h
    end
  end
end

This will produce a simple hash that looks something like this:

Movie.data_for(2, 5, 6, 89, 2467)
# => { 2 => { :id = > '2', :score => '4' }, ... }

You can clean up the structure if you like to convert strings to integers as required, and this will further decrease your storage requirements. You can store a lot of data with plain numbers and simple hashes.

Stashing this hash in Memcache is, at this point, a simple exercise of using Rails.cache