I'd like to differentiate between errors that happen because something is messed up in system (eg. a file is missing) vs. erroneous behavior on behalf of user (eg. selecting too many items in the GUI for some command).
Is there something like "UserMistakeException" (Either in JavaSE or Eclipse API-s) that programmers tend to throw in these cases? I would then present this kind of errors differently.
I'd say exceptions are like their name says, something exceptional and not something that happens during normal use of the program. Normal use will include some possible mistakes done by the end user. User input validation is a totally different issue and you should simply prepare for errors made by the user in a way that doesn't require throwing any Exceptions.
No, there is no such exception. And I do not see an exceptional situation when an user selects something that is selectable.
You'd rather rethink your interaction design. May be you should prohibit selection of too many items. You could allow selection but prevent execution by disabling some buttons.
But tell your user what is going on in every case.
In my application there is a custom exception (extending RuntimeException) which I have created for situations like this. When this kind of exception has been thrown. the client will display the message to the user in an error dialog. All other exceptions cause a generic dialog to be displayed, with Details button, which allows the user to see the stacktrace and send it to me. I don't think Java contains any exception type suitable for this, so you have to create your own.
However, I completely agree with the other answers on that the user interface should prevent so called "user mistakes". Preventing errors is one of the basic rules of usability. If the user selects too many items, the client should validate that, display it in a meaningful way and not throw an exception. In my application the special exception type is only for cases where the UI couldn't validate the data properly before sending it to the server. Actually getting the custom exception is now very rare.
If it is a user mistake, then your program should try to get the user to correct it. (If there is something wrong with the file I chose, then tell me what is wrong and ask me to try again. If I selected too many items, then tell me how many items I need to deselect.)
As a human being user, I am not prepared to catch Exceptions, so don't throw them at me.