If you have internal library errors (unexpected), let the runtime exceptions propagate. If there is a bug in the library code, the client is expected to crash. Fix the bug in the library. Don't catch all and return a library-specific exception, this won't do any good.
If you expect things to (sometimes) go wrong, build this into the API. This is based on the principle that you shouldn't use exceptions for normal program flow.
For instance:
ProcessResult performLibraryTask(TaskSpecification ts)
This way you can have ProcessResult indicate error conditions:
ProcessResult result = performLibraryTask(new FindSmurfsTaskSpecification(SmurfColor.BLUE));
if (result.failed()) {
throw new RuntimeException(result.error());
}
This approach is similar to the return null approach, but you can send more information back to the client.
EDIT:
For interaction which doesn't conform to the agreed protocol, you can throw runtime errors, which you can document. For instance:
if (currentPeriod().equals(SmurfColor.BLUE) && SmurfColor.GREEN.equals(taskSpecification.getSmurfColor()) {
throw new IllegalStateException("Cannot search for green smurfs during blue period, invalid request");
}
Please note that this is due to an interaction which breached the contract, and is not expected to happen.