I've used this for showing locally generated html in the default browser, in my case filename is something like "c:\temp\page.html", perhaps replacing filename with the URL might work??
ShellExecute(NULL,"open",filename,NULL,NULL,SW_SHOWNORMAL);
Updated:
http://support.microsoft.com/kb/224816
How ShellExecute Determines Whether to Start a New Instance
When ShellExecute looks through the registry, it looks for the shell\open subkey. If the shell\open\ddeexec key is defined, then a Dynamic Data Exchange (DDE) message with the specified application IExplore and the topic WWW_OpenURL is broadcast to all top-level windows on the desktop. The first application to respond to this message is the application that goes to the requested URL. If no application responds to this DDE message, then ShellExecute uses the information that is contained in the shell\open\command subkey to start the application. It then re-broadcasts the DDE message to go to the requested URL.
So it looks like you have no control over opening a new window. Whatever browser currently running can handle opening it in whatever way they want.