Re-inventing the Wheel: Yeah, you'd be doing it. But I note OP used the word "minimal" in the problem statement, so assuming it really doesn't need to scale up to all that, it may be a small enough wheel as to not matter. The product I currently work on supports OpenGL on three platforms (Win, Mac, Linux) and we built all our own widgets (text boxes, buttons, dialogs). It's a lot of work but now that we've done it we own a huge chunk of our stack and don't have to debug into third party frameworks when things don't work as expected. It's nice having complete control of the experience. There's always something you want to do that a framework doesn't support. Like everything in our business, it is a trade-off and you just have to weigh your needs against your need to finish on time.
Portability: Yes, you will still have to write platform specific code to boot-strap everything. This will be difficult if you've not done it before as it requires you to understand all the target platforms.
Windows Drivers: We've found that graphics card manufacturers have much better support for DirectX on Windows than OpenGL, since that's what is required to get MSFT certification. Often low- to mid-range cards have bugs, missing functionality, or outright crashes in their OpenGL support.