I have access to ruby's exception hierarchy (it's mentioned in both the pickaxe and the hummingbird), but I'm not sure which exception to use because I haven't found any information on what each of the terms mean.
Does using the right exception class matter?
No.
Or rather, if it doesn't matter to you, it generally doesn't matter to anyone else. If you are writing a library, you should use something the users of your library will understand, but that's about it.
"It depends".
One of the main problems with Ruby is the lack of good specification. It used to be worse, though.
This is mainly a question of style: If your error can be described well with one of the builtin exception classes, use it. If you think a subclass makes sense, use it.
Otherwise, you could consider to follow this lead from a C++ coding standard:
Creating very elaborate exception hierarchies is a waste of time. Nobody ends of caring and all the effort goes to waste. Instead, create one exception per library or namespace and have an exception reason within that exception to indicate the type of the exception.
For example, for your OS encapsulation libary, make an exception called OsencapException.
It matters when creating your own exceptions. One important caveat is that exceptions which inherit from Exception rather then StandardError (common mistake) will not be caught by rescue (without any arguments).