First, you're forgetting something: You need to support 3.2 if you want your app to run on the iPad.
You don't have to "sacrifice functionality"; you just have to code your app to detect what features are available at runtime and fallback gracefully.
If your app isn't free, then the real question is "Will users who don't upgrade buy my app?", and I suspect the answer is generally no. Apparently a substantial portion of the 2.x installed base are iPod Touch users who don't want to pay for the upgrade.
The other factor is the amount of work. If you use few enough iOS 4 features that it's not much work to also make it run on 3.1.3, then supporting 3.1.3 is a no-brainer. Of course, never support an OS version you can't test.