I had a similar problem myself a while ago and did something along these lines (C# .NET 2.0);
public void StreamURLContents(string URL)
{
WebRequest req = WebRequest.Create(URL);
using (HttpWebResponse resp = (HttpWebResponse)req.GetResponse())
using (Stream dataStream = resp.GetResponseStream())
using (StreamReader reader = new StreamReader(dataStream))
{
string currentLine = reader.ReadLine();
while (currentLine != null)
{
Response.Write(currentLine);
currentLine = reader.ReadLine();
}
}
}
You would have to tailor the writing of the HTML to suit your particular application obviously, and you'd break all of the relative links in the target site (image URLs, CSS links etc.), but if you're only after simple text HTML and want your web app to grab it server-side, then this is a good way to go.