Add initialization step to Rails 3 boot process only in server mode

Question

According to http://edgeapi.rubyonrails.org/classes/Rails/Railtie.html, if I write a Rails 3 plugin and I want to hook into the initialization process, I write

class MyRailtie < Rails::Railtie
  initializer "my_railtie.configure_rails_initialization" do
    # some initialization behavior
  end
end

However, this initializer appears to be executed when you run, for instance, a Rails rake task, not just when you run rails s or similar. My question is, how do I prevent my code in this block from being run during Rails tasks, as opposed to full Rails server boot-ups? This seems to be a common problem with Rails 3 plugins.