Unified way to scrape HTML from any type of browser process?

Question

Is there a unified way to do this? Browsers usually don't respond as expected to user32's GetWindowText and SendMessage, which you can use to scrape the text out of most win32 applications.

I'd like to get the equivalent of "View Source" on the open web page.

Currently, I'm using the API for screen readers to scrape from IE, but that doesn't work for FF or Chrome or etc.

ANY suggestions (not requiring me to proxy or try to connect to a web server) to do this for different browsers is GREATLY appreciated. Again, I'm using the screen reader API for IE (which doesn't always work correctly!)...

---

This question now has a bounty.

To collect the bounty, I want to see code, in c#, to scrape an IE window and a FireFox window.

Alternatively, code for a HTTP proxy that does the same thing (i.e., record the current request's content) that works for HTTPS as well.

---

Let me be clear... My requirements are to scrape a window. Oh, and by the way, that window is a browser. I give you the option to show me a proxy to claim the bounty. I already understand there are proxies and what they do, telling me about their existence isn't helping me.

I also can't just connect to the url in code and scrape the response object. Why? Requirements won't let me, among other issues.

I require a solution in .net, as the tags say.

Answer 1

Why don't you just create a simple HttpRequest on the page to get all the html in a String?

Edit:

Here is some code that illustrate the meaning of my first sentence.

Answer 2

I'd have thought it was a per-browser thing.

Answer 3

I would suggest you use a mechanize "module" . It will eliminate all need for browsers ( some of them include JavaScript support,too ) . Here are some links :

• Python Mechanize

• Ruby Mechanize

• Perl Mechanize

Of course there are alternatives for other languages too , for example , Java has HtmlUnit etc. If there is a reason to why you need the output from the browser's window , please elaborate on that .

EDIT : having read your comments , why don't you make the browser go through a proxy ? This way you can monitor all browser traffic .

Answer 4

Not sure if this is exactly what you want, but you could use WatiN (that uses SHDocVw) to control the browser) to grab the source of a specific url that you control from the program. Downside is that to make it work with FF you only get the what's inside the . It would go something like:

        IBrowser browser = BrowserFactory.Create(BrowserType.FireFox);

        browser.GoTo(url);
        browser.WaitForComplete();

        src = browser.Html;

Answer 5

You could use Selenium (cross-browser) to collect the data you need directly from the DOM tree via id/CSS/XPath locators. That way, you would have access to DHTML content (which can't possibly be scraped by just parsing the fetched HTML). You can either issue a runScript command or write your own Selenium extension command to post the collected data (JSON via Ajax) to a URL. Or if you don't want to deal with Ajax, you could change the <title> and grab the window title of the running browser window.

Most answers here focus on directly scraping the fetched HTML. However, since most modern websites employ DHTML in one form or another, wouldn't you want to scrape the content off the rendered page?

Answer 6

I can see some reasons why your requirements require you to get the HTML from within the browser rather than scrape separately with Mechanize, Scrapy, etc.:

• You don't know ahead of time what page you want to scrape
• You need to login first and can't easily automate it

You could try to produce a solution at the Win32 level that works with any browser but that will be specific to Windows and probably contain a lot of code to deal with differences between the different browsers.

How about writing a Firefox extension? This would easily allow you to extract the current page's HTML on-demand. I'm sure how well it will apply, but take a look at Piggy Bank. One additional benefit of a Firefox extension is that its cross-platform. I'm not too familiar with Google Chrome, does it allow similar extensions?

Answer 7

Hmm. Just a thought, Add a FileSystemWatcher on known temp-folders for all browsers cachestorage. When Filesystemwatcher signals that a new page has been cached, read the file in an own instance of internet explorer in your program, fetch the outertext or what you want from the DOM to get all text but not the html-tags.

Possible pitfall is if secure pages dont land in the cache, I think they do but are erased after the visit, but Im not sure..

Answer 8

Can't be done in purely managed code. You can do this in something like autohotkey.

Select window with title xxx send: ctrl + a ctrl + c

and paste where you need it.

Answer 9

This is strictly for viewing/displaying in IE but you could use the WebBrowserControl.

    public partial class Form1 : Form
{
    public Form1()
    {
        InitializeComponent();
        Scrape();
    }

    private void Scrape()
    {
        this.webBrowser1.Url = new Uri("http://www.stackoverflow.com");
        this.webBrowser1.DocumentCompleted += new WebBrowserDocumentCompletedEventHandler(webBrowser1_DocumentCompleted);
    }

    void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
    {
        if (e.Url == webBrowser1.Url)
        {
            MessageBox.Show(webBrowser1.DocumentText);
        }
    }
}