About the math: You should be both fluent with trigonometric functions and being able think in vector space.
You might want to start doing some 2d-games first. I have recently played with a small Lua based framework called Löve, which is really neat and allows you to produce first results really fast. It might however be limiting of you want to make something more complex.
If you want to go straight to 3d, then I would recommend you some high level environment first. Not having to worry about memory or shooting yourself with C++ in the foot all the time does quite help if you are learning a new skill.
XNA is an interesting framework, I've recently played with NeoAxis, which does also seem quite nice. These are both based on .NET. XNA only works within the Microsoft ecosystem, but you can also make Xbox/Zune/Windows Phone games with it. NeoAxis does have support for other operation systems with mono.
There are other high level environments like Unity, but that costs a bit.
Some friends of mine also recommended trying out the blender engine to me.
There are also commercial engines that support free mods or even non commercial standalone games. For example the Unreal Engine with the UDK, the Source engine (HL2 etc) or maybe the Cryengine (Farcry 1, Crysis). I have worked only with the source engine, which is quite nice but its documentation is really bad.
Of course you can also take the hardcore path and start either building your own complete game engine, by integrating together different libraries, like OGRE for graphics, ODE for physics, raknet for networking etc, or even digging up all the theory and implement all that technology yourself. Do whatever pleases you most.
I'd also recommend you reading articles on pages like beyond3d, Gamasutra and watching some presentation from the GDC Vault.