HttpWebRequest reades only homepage

Question

Hi I tried to read a page using HttpWebRequest like this

    string lcUrl = "http://www.greatandhra.com";
    HttpWebRequest loHttp = (HttpWebRequest)WebRequest.Create(lcUrl);
    loHttp.Timeout = 10000;     // 10 secs
    loHttp.UserAgent = "Code Sample Web Client";
    HttpWebResponse loWebResponse = (HttpWebResponse)loHttp.GetResponse();
    Encoding enc = Encoding.GetEncoding(1252);  // Windows default Code Page
    StreamReader loResponseStream =
       new StreamReader(loWebResponse.GetResponseStream(), enc);
    string lcHtml = loResponseStream.ReadToEnd();
    mydiv.InnerHtml = lcHtml;
   // Response.Write(lcHtml);
    loWebResponse.Close();
    loResponseStream.Close();

i can able to read that page and bind it to mydiv. But when i click on any one of links in that div it is not displaying any result. Because my application doesnt contain entire site. So what we will do now.

Can somebody copy my code and test it plz

Nagu

Answer 1

The code you are putting inside .InnerHtml of the div contains the entire page (including < html >, < body >, < /html > and < /body> ) which can cause a miriad of problems with any number of browsers.

I would either move to an iframe, or consider some sort of parsing the HTML for the remote site and displaying a transformed version (ie. strip the HTML ,BODY, META tags, replace some link URLs, etc).

Answer 2

I'm fairly sure you can't insert a full page in a DIV without breaking something. In fact the whole head tag may be getting skipped altogether (and any javascript code there may not be run). Considering what you seem to want to do, I suggest you use an IFRAME with a dynamic src, which will also hopefully lift some pressure off your server (which wouldn't be in charge of fetching the html to be mirrored anymore).

Answer 3

If you really want a whole page of HTML embedded in another, then the IFRAME tag is probably the one to use, rather than the DIV.

Rather than having to create a web request and have all that code to retrieve the remote page, you can just set the src attribute of the IFRAME to point ot the page you want it to display.

For example, something like this in markup:

<iframe src="<%=LcUrl %>" frameborder="0"></iframe>

where LcUrl is a property on your code-behind page, that exposes your string lcUrl from your sample.

Alternatively, you could make the IFRAME runat="server" and set its src property programatically (or even inject the innerHTML in a way sismilar to your code sample if you really wanted to).

Answer 4

But when i click on any one of links in that div it is not displaying any result

Probably because the links in the download page are relative... If you just copy the HTML into a DIV in your page, the browser considers the links relative to the current URL : it doesn't know about the origin of this content. I think the solution is to parse the downloaded HTML, and convert relative URLs in href attributes to absolute URLs

Answer 5

If you want to embed it, you need to strip everything but the body part. That means that you have to parse your string lcHTML for <body....> and remove everything before and includeing the body tag. You must also strip away everything from </body>. Then you need to parse the string for all occurences of <a href="....."> that do not start with http:// and include h t t p://www.greatandhra.com or set <base target="h t t p://www.greatandhra.com"> in your head section.

If you don't want to embed, simply clear the response buffer and stream the lcHTML string back to the browser.

PS: I had to write all h t t p with spaces to be able to post this.

Answer 6

Sounds like what you are trying to do is display a different site embedded in your site. For this to work by dropping it into a div you would have to extract the code between the body tags as it wouldn't be valid with html and head in the middle of another page.

The links won't work because you've now taken that page out of context in your site so you'd also have to rewrite any links on the page that are relative (i.e. don't start with http) to point to a page on your site which will then fetch the other sites page and display them back in your site, or you could add the url of the site you're grabbing to the beginning of all the relative links so they link back to that site.