I've a running nginx and Ubuntu 8.09 site running a rails app -- but we are not making use of Peassenger anymore and it seems to take up a lot of memory.
I google around and there's tons of info on how to install Passenger, but I didn't find any on uninstalling.
Edit your httpd.conf and remove the lines you once added to install mod_rails.
If you don't remember, look at the installation tutorials and undo the steps in reverse order.
IF your only concern is the memory usage, removing the Passenger lines from the webserver config file will cause it to no longer be loaded.
If you want to completely remove it from your system, them you'll have to uninstall the gem too, assuming that's how you initially got it.
A tip for the future: I generally keep lots of tiny config files, one for each module that I'm using, so that it's easy to find, edit, and eventually, remove them. This is not the general case, though.
edit: hum, apparently Passenger compiles a new nginx with support for it, so it's not as simple as removing a module. Well, in that case I'd recommend you wipe your current nginx binary and compile a new one without mod_rails. Notice there may still be lines in the existing config file to be removed.