I'm not aware of any libraries that satisfy your request (at least not unless I decide to publish the code for my own tracer...).
Writing a tracer isn't actually that hard anyway. I'd strongly recommend getting hold of a copy of "An Introduction to Ray Tracing" by Glassner. It goes through the actual math in relatively easy to understand terms, and also has a whole section on "how to write a ray tracer".
In any event, a "library" isn't all that much use on its own - pretty much every ray tracer has its own internal libraries but they're specific to the tracer. They typically include:
- a base class to represent 3D objects
- subclasses of that for each geometric primitive
- vector and matrix classes (3D and 4D)
- texturing functions and/or classes
- light classes of various types (point light, spot light, etc)
For my own tracer I actually used the javax.vecmath
packages for #3 above, but had to write my own code for #1 and #2 based on the Glassner book. The whole thing is well under 2k lines of code, and most of the individual classes are about 40 lines long.