How are you specifying your angle of rotation? in Degrees or in Radians?
Sometimes I get results like those you describe when I've accidentally specified my angle in degrees when it should've been radians (and as a result the object has been rotated several hundred degrees instead of a few)
I don't know for sure given your specific example, but its worth suggesting.