A: 

You could implement your own tool using reflection, especially if you want to examine distinct changes of your code. It also works with bytecode.

PartlyCloudy
+1  A: 

Take a look at japitools. These tools are used by the GNU Classpath project to compare their APIs for signature compatibility with different versions of the Sun Java class libraries.

Stephen C
A: 

I've used the JDiff doclet in the past for this. Depending what you're doing it can be a pain to setup but I think the Ant task works fairly well.

Alex Miller
+2  A: 

Clirr is a tool that checks Java libraries for binary and source compatibility with older releases.

Apache Commons use it, it creates reports like this (for Commons Lang).

Thilo