Since Swing paints all UI itself, it is not really influenced by Mac OS. There are exceptions, though. If you allow Java to use a native menu, the menu will behave like any other Swing menu, but it is in fact translated to native menu functions (so it will be displayed on top of screen and behave like a native Mac OS X menu... nothing you as programmer need to care about, but great for Mac users, who hate menus to appear elsewhere).
I guess your issue is probably something else, but it's hard to guess, if I have no code to work with. Can you break down the issue into an ultra minimal test case? E.g. can you just draw a JFrame, add a singe one of your custom buttons there, that shows the problematic behavior and just use as much code as is necessary to paint this minimal UI and expose the problem? If you'd post this code here, I'd take a closer look into it and will try to find out what causes the issue and how to resolve it (I'm a graduated Java programmer, but I also have years of Mac OS X development experience; so these are two fields of knowledge I should know really well, but I usually don't combine them, as I don't develop in Java for Mac OS X).