As MSN said, they are used in different cases. CGAffineTransform is used for 2-D manipulation of NSViews, UIViews, and other 2-D Core Graphics elements.
CATransform3D is a Core Animation structure that can do more complex 3-D manipulations of CALayers. CATransform3D has the same internal structure as an OpenGL model view matrix, which makes sense when you realize that Core Animation is built on OpenGL (CALayers are wrappers for OpenGL textures, etc.). I've found that this similarity of internal structure, combined with some nice helper functions that Apple provides, can let you do some neat OpenGL optimizations, as I post here.
When it comes down to choosing which do use, ask yourself if you're going to work with views directly in a 2-D space (CGAffineTransform) or with the underlying Core Animation layers in 3-D (CATransform3D). I use CATransform3D more frequently, but that's because I spend a lot of time with Core Animation.