Are there greater chances of my app getting rejected by Apple if I use a custom category to extend the functionality of an UIKit interface element, let's say UIButton?
Clarification: I'm thinking about implementing a toggle UIButton, that will alternate between pressed/released states on each tap.
That sounds like what categories are for... You're extending the UIButton, which is a standard OO technique. As long as your category doesn't break any rules itself (i.e. accessing undocumented internal code) I'd think you'd be fine, subject to the standard disclaimers about nobody knowing why Apple does the things it does at times...
You can use custom categories to do whatever you want. If you include a private API, chances are you will not be rejected, but you may find yourself in a pickle if they change the API. It is good practice to wrap any methods that use a private API in a @try block, and include a workaround in case the worst happens
Of course you can extend the functionality of UIButton through categories or by sub-classing.
HIG violations are the most common cause of app rejection and that's what I'd be concerned about. But that's your judgement call. The worst that can happen is that they ask you fix it.