Actually, this problem is even worse on the iPhone 3G, which can't even manage to display two full size photos (taken with it's own camera) at the same time.
If you can make the photos smaller ahead of time by resizing them (server side if you're snagging them from your own services, or client side if they're somehow generated on the device or retrieved from third party services), you should. They will load faster, and you can put more of them on the screen at once without running into memory issues.
The Three20 library has a really great photo-browsing control for the iPhone. I'm pretty sure they only have three photos around at any given time... the photo you're looking at, and the photos going forward or backwards.
http://github.com/facebook/three20
I also think there is an iPad branch of the Three20 controls (at ./tree/ipad), although I don't know if they include the photo browser or not, and haven't tried them yet myself... maybe that's an option for you if you don't want to spend a lot of time on that particular feature of your application. If the iPad photo browser isn't done maybe you could just adapt the iPhone version to suit your needs (shouldn't be hard).
If you want to bake the feature yourself, I'd take some inspiration from the UITableViewController. Your controller should come with associated datasource and delegate protocols to retrieve photos and respond to user events. The controller base class itself should reuse three UIImageViews and should shuffle those around to create the illusion that the user is scrolling through a large list.