First, some general comments about you requirements:

- reading OBJ or OFF files is very easy. You could implement it yourself, on top of a library providing the more geometric features, in a few minutes. On the other hand, the geometric part of such libraries is so much more tricky that you should certainly focus on your requirements which really deal with the geometric algorithms, and try to find something which suits your needs. Then, of course, if there is a tie, start considering this interface issue.
- in terms of geometric operations, you ask for intersection. Do you mean primitives intersection ? (for which good and simple algorithms can be found and implemented) or computation of the intersection of two meshes ? or collision detection ? (which are delicate questions, with no simple answer)
- if you are more specific, from a higher level point of view, about the kind of tools you want to build, then people will be able to direct you to the right tool. Your requirements are too low-level.

As far as I understand your question, it seems to me that you do not clearly see the point of libraries like CGAL and OpenMesh. Such libraries may not provide all the higher level tools you need, but their aim is to provide you (especially in the CGAL case) all the geometric framework upon which you can build a geometric application. Such geometric frameworks are very delicate to design, especially because of the robustness issue, which is very specific to computational geometry. And without such a framework, building a robust application is an horrendous effort.

If you do not find a library which suits your need, you should seriously consider using a library such as CGAL as the underlying framework for your development. It will prevent the appearance of the robustness related problems, that you will typically only start noticing late in your development process, when changing the underlying framework will be painful. As an aside, CGAL has an extensive documentation, and a very active users' mailing-list.

If you do not know about robustness issues in geometry software, have a look at this page:
robustness issues