What's happening is that your Application and/or ApplicationSpawners are shutting down due to time-out. To process your new request, Passenger has to startup a new copy of your application, which can take several seconds, even on a fast machine. To fix the issue, there are a few Apache configuration options you can use to keep your Application alive.
Here's specifically what I've done on my servers. The RailsSpawnMethod and RailsAppSpawnerIdleTime are the configuration options most important in your situation.
# Speeds up spawn time tremendously -- if your app is compatible.
# RMagick seems to be incompatible with smart spawning
RailsSpawnMethod smart
# Keep the application instances alive longer. Default is 300 (seconds)
PassengerPoolIdleTime 1000
# Keep the spawners alive, which speeds up spawning a new Application
# listener after a period of inactivity at the expense of memory.
RailsAppSpawnerIdleTime 0
# Additionally keep a copy of the Rails framework in memory. If you're
# using multiple apps on the same version of Rails, this will speed up
# the creation of new RailsAppSpawners. This isn't necessary if you're
# only running one or 2 applications, or if your applications use
# different versions of Rails.
RailsFrameworkSpawnerIdleTime 0
# Just in case you're leaking memory, restart a listener
# after processing 5000 requests
PassengerMaxRequests 5000
# only check for restart.txt et al up to once every 5 seconds,
# instead of once per processed request
PassengerStatThrottleRate 5
By using "smart" spawning mode and turning off RailsAppSpawnerIdleTime, Passenger will keep 1 copy of your application in memory at all times (after the first request after starting Apache). Individual Application
listeners will be fork
ed from this copy, which is a super-cheap operation. It happens so quickly you can't tell whether or not your application has had to spawn a listener.
If your app is incompatible with smart spawning, I'd recommend keeping a large PassengerPoolIdleTime and hitting your site periodically using curl and a cronjob or monit or something to ensure the listener stays alive.
The Passenger User Guide is an awesome reference for these and more configuration options.
Finally, but unrelated to your question, consider using Ruby Enterprise Edition if you go with Smart spawning. It can save you a lot of memory.