I created a 57X57 Circle-shaped icon (without shine) which has transparency outside of the circle-shape for my app. I can successfully install the app on Simulator and iPhone. It works fine, and looks great.
However, can I submit an icon with transparency to Apple. Will they accept it? I can't find any apps which have transparency in their icon... instead they use a kind of pseudo-transparency - a jet black background with no shine.
For the app, you can use your transparent PNG as the icon. When you upload to the itunes store, they require a .jpg which of course has no transparency.
As for if they will accept your app with a substandard shaped icon, maybe, maybe not. The could approve it one time and then reject an upgrade, it will probably depend if they are paying attention and notice the icon or not.
Make all of your transparent areas black. Then add a UIPrerenderedIcon key with a value of "YES" to your applications Info.plist.
Otherwise it will appear shiny on the users phone.
I've looked into this for a client who would like a circular transparent icon to match his company's logo. Here's what I've found:
The 512x512 iTunes icon is uploading in JPEG or TIFF format, which don't support transparency, so you'd have to make up a square background for your circle. Since the home screen icon is supposed to look exactly like the large icon, your home screen icon will also have to have the same background.
As brandon suggests you could use a pre-rendered icon with a black background, but you'd be at the mercy of the App Store reviewers. The docs say your home screen icon must be a "flattened, square-image icon that is 57x57 pixels" (my emphasis). You might get away with a circular icon with a black background, but it really depends on how the reviewer tests that requirement. You might get away with it if they don't check, or only check that the alpha channel is 100% everywhere, but won't if they decide based on whether the icon looks square.
A circular icon will look odd next to all the other icons on the users' phones, so it's probably a bad idea anyway.
Apple may have updated the interface guidelines since this question was asked, but transparency is now expressly not allowed. See more here: http://developer.apple.com/iphone/library/documentation/UserExperience/Conceptual/MobileHIG/IconsImages/IconsImages.html