The other downside is that Visual Studio for x64 applications has some limitations:
Edit and Continue is not available
for 64-bit debugging.
You cannot debug in mixed-mode, calls
from native code to managed code, or
vice versa, in 64-bit code.
See: http://msdn.microsoft.com/en-us/library/ms184681(VS.80).aspx
Note: The 64 bit compilers aren't installed by default. You have to select them during the install.
I also just found this (because I'm optimising a x64 application myself).
"Porting and Optimizing Applications on 64-bit Windows for AMD64 ..."
http://download.microsoft.com/download/5/b/5/5b5bec17-ea71-4653-9539-204a672f11cf/AMD64_PortApp.doc
Has lots of good pointers on compiler switches and the like.