Probably because extending it and overwriting some of it's methods would probably break it. And making it easier to overwrite methods would expose to much of the inner workings, so if in the future they decide to change those (for performance or some other reasons), it would be harder for them to change the class without breaking all the classes that extend it.
For example, consider the following method in the class:
public boolean nextBoolean() {
clearCaches();
return Boolean.parseBoolean(next(boolPattern()));
}
Say you want to overwrite this because you want to make 'awesome' evaluate to a 'true' boolean (for whatever reason). If you overwrite it, you can't call super.nextBoolean(), since that would consume the next token using the default logic. But if you don't call super.nextBoolean(), clearCaches() won't be called, possibly breaking the other not overwritten methods. You can't call clearCaches() because it's private. If they made it protected, but then realized that it's causing a performance problem, and wanted a new implementation that doesn't clear caches anymore, then they might break your overwritten implementation which would still be calling that.
So basically it's so they can easily change the hidden parts inside the class, which are quite complex, and protecting you from making a broken child class (or a class that could be easily be broken).