modifying Rails load paths in the Rails::Configuration block

Question

I'm trying to figure out how to have Rails NOT load the root models and controllers at app/models and app/controllers at boot. I've tried setting config.load_paths (with config being the Rails::Configuration instance) to both an empty array and nil, but those paths still show up within $LOAD_PATH.

Am I missing something here, or is what I'm trying to do not possible?