I am looking for book titles or other resources for showing me how to get started and become advanced at windows based software with 3d graphics.
I used to read a book about beginning DirectX, i didn't finish it but it looked interesting and easy to follow. Also if you'll want to use a developed game engine, instead of creating your own, you can have a look at Ogre, it's free, open source and it has a lot of tutorials and examples, with a huge community behind. If you want something more simple and to use .Net you can have a look at Microsoft XNA.
That depends on the application. A modeling application (Maya, Rhino) works differently from a game engine (Unreal, Crysis), a renderer (RenderMan, VRay) or a specialized tool such as Z-Brush. The math basics are the same, but data structures largely depend on the application.
If you just want to get some 3d graphics on the screen, go for XNA. It's focused on games, but it should get you started.