The "flixster" app in the appstore only supports landscape and landscape-upside down views, so clearly (along with hotpaw2's answer) at least two people have got away without supporting all the orientations.
Much like anything else to do with app store reviews, I suspect you may or may not get away with it, depending on how the reviewer feels. The documentation says:
"With few exceptions, applications
should support all interface
orientations on iPad devices."
which would tend to suggest there's a risk (albeit perhaps a small one) that your app will get rejected.