+1  A: 

No, you're pretty much doing it the best way possible. I don't think there is a built-in DirectX function that does what you want. For step 4, you can use D3DXMatrixRotationAxis(). Just be careful about the edge cases, such as when |A| or |B| is zero, or when the angle is 0° or 180°.

Adam Rosenfield
+1  A: 

It's probably more of a typo than a thinko, but acos(A.B) is the angle, not its cosine. Similarly for point 2.

You can calculate the sin from the cos using sin^2 + cos^2 = 1. That is, sin = sqrt(1-cos*cos). This would be cheaper than the vector expression you are using, and also eliminate the special cases for 0/180 degrees.

Hugh Allen
I am not sure that calculating the sine from sqrt(1-cosine*cosine) will work. This is because cosine*cosine will cause us to lose the sign of the cosine
Vulcan Eager
You don't "lose the sign of cosine" because you use the cosine as-is in the rotation matrix. What you should really ask is do you get the right sign for sin? In this case (0-180 degrees) you only want the positive square root, so it's all good.
Hugh Allen
I agree completely!
Vulcan Eager
Hugh, What about the 0 to -180 range?
Vulcan Eager
Negative angles aren't necessary for this particular application. When rotating from B to A the axis vector will be negated rather than the rotation angle.
Hugh Allen
That's how things ended. Thanks again for a very prompt reply.
Vulcan Eager
+1  A: 

You might look at the following article from siggraph link text

Adam Tegen
A: 

Maybe you can use D3DXMatrixLookAtLH ?

greg2fs