How to screen scrape HTTPS using C#?
+4
A:
You can use System.Net.WebClient to start an HTTPS connection, and pull down the page to scrape with that.
Aequitarum Custos
2009-12-04 15:31:56
And if you need to log in to get the https content?
Oded
2009-12-04 15:35:34
You'll need to make sure you assign a CookieContainer in the WebClient for cookies to be passed across multiple requests (eg. the login page and then the content page).
Danny Tuppeny
2009-12-04 15:38:41
The site is using URL rewriting.How do i get the complete url?
Jignesh
2009-12-04 15:48:51
If you're talking about server side URL rewriting, no idea. But if you're talking about javascript, simply parse it in code.
Aequitarum Custos
2009-12-04 16:24:24
+1
A:
If for some reason you're having trouble with accessing the page as a web-client or you want to make it seem like the request is from a browser, you could use the web-browser control in an app, load the page in it and use the source of the loaded content from the web-browser control.
Cyril Gupta
2009-12-04 15:34:00
+4
A:
You can use System.Net.WebClient to grab web pages. Here is an example: http://www.codersource.net/csharp_screen_scraping.html
Arriu
2009-12-04 15:35:10
link dead: i think this may be the updated link - http://www.codersource.net/microsoft-net/c-advanced/html-screen-scraping-in-c.aspx
Simon_Weaver
2010-10-20 22:03:08
A:
How can i start an HTTPS connection pull down the secure page by using System.Net.WebClient class.
Ajit
2010-03-19 11:02:52