Backgrounded apps can be killed on a memory warning to give the foreground app more memory, so the whole concept of "memory usage" and "free memory" is somewhat useless. If you watch the app/console log and don't see any memory warnings, then stop worrying. "Level 1" memory warnings aren't surprising if you're doing anything vaguely memory-heavy (taking pictures, AVCapture, image processing). "Level 2" memory warnings mean you should be a bit careful, and "level 3" is the last one before your app gets killed IIRC.
Test on the worst device your app supports, e.g. the original iPhone/iPod Touch running 3.1.3 or the iPhone 3G/iPod Touch 2G on 4.0.1 if you don't support 3.1. Testing on an old device also reveals performance issues, especially where graphics hardware is involved — CALayer.mask, CALayer.contentCenter, and UIView.contentStretch are only properly supported on the 3GS and above (iPhone 4, iPad, presumably iPodTouch 3G), and either render a bit odd or sluggishly on older devices. It's up to you to decide what's "acceptable" performance for older devices.
It's not usually worth testing on old OSes for memory/performance reasons (there was a syslogd memory leak in 2.something, but you can assume that most users have upgraded). Note that newer OSes are often slower — 4.0 is much slower than 3.1.3 on a 3G, but performs perfectly well on a 3GS. Perhaps it's ARM vs. Thumb, or just planned obsolescence, but it's worth knowing about.