I'd like to get an idea what kind of math is typically necessary for 3D game engine programming? Any specific math (such as vector geometry) or calculation algorithms (such as fast fourier transforms), or is this all abstracted away with DirectX/OpenGL so that highly complex math isn't really necessary any more?
Matrices, trig, geometry mostly and a bit of linear algebra
Take a look here http://www.essentialmath.com/
Complex math comes in to play, but most important is an understanding of the concepts behind such math and often not the math itself. So long as you understand how it all comes together, there are often helper methods for many of the calculations you will need. Of course that depends largely on the development platform you are using as well.
For the most part linear algebra and computational geometry; Quaternions are used extensively and numerical analysis is necessary if you are working on a physics engine.
How much you'd use the stuff on a daily basis depends on what you are doing. If you are a graphics programmer, and therefore building the 3D graphics engine itself, then you would likely be implementing or maintaining implementations of class libraries and functions that actually do the math, so it would be relatively important to know the gory details. If you are using the library as client or working on another part of the game engine (AI, audio, UI, camera, etc.) then you will need to understand the math conceptually but you can certainly get away with not knowing how to implement a matrix inverse on a whiteboard off the top of your head.
Graphics APIs do not eliminate the need for someone to know this math; they are limited to drawing, so all the scene management and world simulation needs to be implemented outside the graphics API. Of course, there are middleware options there too but many studios roll their own systems.
There are a fair number of resources targeted at this market. For example, on Amazon there are books like 3D Math Primer For Graphics and Game Development, and there's probably a lot of stuff online too.
Linear Algebra! Lots of lots of Linear Algebra!
Here are just classes and example situations where you need them
- Vector - position, velocity, normals
- Matrix - transformations
- Quaternion - rotations (great for bone animations)
- Ray - projectile collision detection
- Plane - projectile collision detection
- Frustum - render culling
- Sphere - render culling, fast collision test
- Axis-Align Bounding Box - culling, collision tests, spacial partitioning
- Oriented Bounding Box - collision tests
- Convex Hull - collision, spacial partitioning
- etc.
You should start with Vector and Matrix as they will be used everywhere in the engine (graphics, physics, AI, etc.)