What is the best way to implement license enforcement in a Java application, so that the application cannot be "distributed" and used without the user purchasing the license?
I suggest you don't even attempt to add copy protection. You will just annoy people. You are much more likely to get sales if people like your product. Being petty and treating customers like thieves is not going to help. (Unless the product really sucks, and you have to prevent people from being able to try it before they buy.)
You might want to send a license key the sole purpose of which is to satisfy users that want to receive something for their money. Just add a status somewhere to show whether it has been paid for. The status needs to be reasonably obvious so that it is clear that something has happened (a product I worked on along time ago came on floppy two discs - half of the support calls were because the installer didn't ask for the second [win32s] disc).
Take a look at truelicense which is an open source framework for doing just that. If you want to make your own there is some documentation that explains the concepts and you can probably get some good ideas by looking at the code as well.
edit: worth mentioning that it is not a copy-protection scheme
Personally I think IntelliJ gets it right: fully-featured version for 30 days. You shouldn't be any more restrictive than that.
The "fully featured" part is important. If your unregistered version is so crippled as to be useless, how do you expect people to decide to buy it? They have to try features before they decide they like them and want them but the notion some vendors have that people will buy something on the promise of features they haven't been able to try is ludicrous.
Oracle is at the most liberal end of the spectrum. You can basically download a fully-featured version of their software and just use it. They treat licensing as a social problem and there's a lot of merit to that argument.
You should never nag or otherwise annoy your potential customers (beyond possibly expiring the software after a predetermined period). The most notice you should give is "X expires in N days" on say a splash screen or something.
The other thing you can and should do is provide a low barrier to entry for your software.
Provide a cheap (if not free) academic or personal version. Marketing is also a good idea. By this I mean take IntelliJ: it's offered free to approved open source projects.
As for the technical aspects of enforcement: don't be too concerned about the technical aspects. Never choose any scheme that annoys your users or, worse yet, slows down your software (as some schemes do) or (even worse) violates the security of the user's operating system (eg like the Sony rootkit fiasco). Your enforcement is not meant to be hack-proof. If people want to steal your software they will. The enforcement system is there as a gentle reminder (and nothing more) for a social not a technical problem.
There isn't a licensing scheme that I am aware of that is not easily breakable. I will not go into specifics, but as an example, in the Java case if it is done in Java then it can be decompiled (as can C++ or any compiled language, but Java is easier than C++ for instance). If it is done natively then you need native methods, and the dll/so could be replaced with one that allows access.
For the C/C++ example I know someone who decompiled the CL (Microsoft compiler) to figure out how they do name mangling. All they had was the binary, and they were able to do it with a disassembler and a lot of time.
Don't bother with this type of thing, it annoys honest users and moderately slows down those that want to copy it. Take a look at World of Goo (great game!) and their take on DRM (which is a similar idea to what you are talking about).
The developers of World of Goo found that their product was pirated 90% of the time (they explain how that number could be inaccurate), contrast that to another game that shipped with DRM and was pirated 92% of the time.
If you are talking about legitimate businesses then they want to be in license compliance usually, so ensuring that they are not copying is a waste. If you are talking about people who would not buy the product to begin with, or just don't want to pay for it, you are not really going to alter the number of people that get your software illegally, you will only slow down the first person who gets it - adter that the cat is out of the bag.