What are installers like the installer for Visual Studio made in? How do they make such a nice installation experience? I know InstallShield can emulate this, but does Visual Studio use InstallShield, or did InstallShield copy it?
I believe VS uses WiX. Not sure, though. There is a movement within Microsoft to standardize on WiX (e.g., it's used for the complex Office installers), but I'm not 100% sure if VS has been converted over yet.
The setup for Microsoft Office 2007 (and probably 2010) was built on top of WiX (source). Note that this does not mean you can emulate the Office 2007 setup yourself using purely WiX.
Aside from what you actually see - Microsoft Office, Visual Studio or any almost any other application designed for Enterprise deployment relies on Windows Installer technology. You can use Installshield, WiX or a variety of other software products to achieve the same end result.
HOWEVER note that Office, Visual Studio, and a whole bunch of other installations use Windows Installer (MSI) without using the Windows Installer UI (UX). One of the awesome things about Windows Installer is that you can build your own external UI handler to make the "visible" things appear how you want, while leveraging Windows Installer and MSI behind the scenes (Office installs and configures lots of MSI packages while appearing as a single setup process to the user).
Generally you would develop your setup UX in native code ideally with minimal dependencies (e.g. no managed languages and depend on older libraries if possible. Users might not have VS2005/2008 dependencies when running setup for the first time so target RTM not SP1). Unfortunately it's not a simple wizard or a library you can depend on, this is effectively developing a small stand-alone application from scratch in a language you may or may not be familiar with (Native code developers are becoming harder and harder to find, managed code is more commonly creeping into setup because it's "easier").
tl;dr - Doing the non-UI portion is a solved problem, use Windows Installer technology (e.g. WiX). The user interface will require custom development, probably in native C++ code.