I personally think the hyphen as word separator makes for maximum readability and typability in general, so I recommend that where possible (in some contexts, a hyphen can't be used, such as in identifiers in most languages). One important thing to bear in mind is that the scheme you pick will have a bearing on the require statement that users will use with your lib, and you want to avoid having a different gem name than library name.
Bad
# gem install my_cool_lib
require 'my-cool-lib'
# gem install MyCoolLib
require 'my_cool_lib'
Good
# gem install my_cool_lib
require 'my_cool_lib'
# gem install my-cool-lib
require 'my-cool-lib'
Unfortunately, a small handful of libraries violate this simple usability rule. Don't be one of those libraries. :)