(Updated to address each of the requirements individually)
If you're looking to do build an MSI package, go and read The Definitive Guide to Windows Installer before selecting an application. Most commercial options hide the underlying structure from you, which means a quicker turnaround if your package is really simple. But it also means it's a real pain in the ass to debug if something goes wrong.
Read the book first, then you'll know enough to understand WiX. Or if you want a WiX GUI then try MSI Factory
While you can use VBS or various other scripting languages with MSI packages, you're better off writing a custom action in C++ for reliability. See why VBScript (and Jscript) custom actions suck for more info.
Easy in WiX to define your own, or there are a number of predefined dialog libraries you can use.
Best solution is to write a C++ DLL to do the validation here, or do product key validation in the software application itself.
For a .NET bootstrapper, you're going to need something else or possibly build your own. Personally I use IRMakeBootstrap that is included with MSI Factory, as is best suited to my needs. It was the most flexible one I could find and purchasing a license worked out cheaper than rolling my own.
Edit: If you're looking for a non-MSI solution, then I'd suggest NSIS. However anyone looking to deploy your software in a corporate environment will almost certainly require an MSI package.