I need JavaScript working almost perfectly in my application that should be able to access web content and let user take a control of it. However, webbrowser component pops new windows in separate instance / application of Internet Explorer and dotBrowser doesn't work with JavaScript properly.
Please, give me some advice - I'd even take some working example and enhance it with function I need it for.
Give a look to GeckoFX, is on open-source component for embedding the Mozilla Gecko (Firefox) in .NET applications.
I could be mistaken, but I believe new windows can be controlled (or at least suppressed) using the Web Browser control in .NET. Have a look at the NewWindow event.
I have had experience embedding both WebKit (Apple Core, used in Chrome) and XULRunner (Gecko) cores in .NET winforms applications. Let me give you some advice:
GeckoFX is your best bet. It is licensed under MPL and it works pretty much out of the box. Follow the instructions carefully and pull the exact XUL version that is indicated. I would say it works better than the WebBrowser control. There is some talk of a significant delay when loading XUL for the first time but I've found it to be negligible. Branding and the like is fully customizable via external files.
WebKit is embeddable with SwiftDotNet. If all you need is solid javascript performance then this may be an option. If you need to be able to download files, then you have some work cut out for you.
Your downside with both routes is a nasty ~7-8mb overhead, even with compression (although I was able to get 6.5mb with lz compression in NSIS). If you are not creating distributable consumer-facing software then this may not be a problem for you.