How do I download a webpage into a stream in .NET

Question

I know this should be a basic question but I am hitting a brick wall. I am looking to go to a URL/URI download the resulting string as if I had opened a file and then get it out into a String variable.

I have been stuffing about with IO.Stream and Net.httpxxx but haven't managed to get the elements to line up in the right way.

I get "the given path's format is not supported" from opening the page in the standard stream, because it's not in the local file system ... that bit i understand, the bit I don't get is ... how do I achieve the equivelent of:

Public Function GetWebPageAsString(pURL As String) As String
        Dim lStream As IO.StreamReader = New System.IO.StreamReader(pURL)
        Return lStream.ReadToEnd

End Function

Answer 1

This function downloads any URI to a file. You could easily adapt it to put it into a string var:

public static int DownloadFile(String remoteFilename, String localFilename, bool enforceXmlSafe)
{
// Function will return the number of bytes processed
// to the caller. Initialize to 0 here.
int bytesProcessed = 0;

// Assign values to these objects here so that they can
// be referenced in the finally block
Stream remoteStream = null;
Stream localStream = null;
WebResponse response = null;   

// Use a try/catch/finally block as both the WebRequest and Stream
// classes throw exceptions upon error
try
{
 // Create a request for the specified remote file name
 WebRequest request = WebRequest.Create(remoteFilename);
 if (request != null)
 {
  // Send the request to the server and retrieve the
  // WebResponse object 
  response = request.GetResponse();
  if (response != null)
  {
   // Once the WebResponse object has been retrieved,
   // get the stream object associated with the response's data
   remoteStream = response.GetResponseStream();

   // Create the local file
   if (localFilename != null)
    localStream = File.Create(localFilename);
   else
    localStream = new MemoryStream();

   // Allocate a 1k buffer
   byte[] buffer = new byte[1024];
   int bytesRead;

   // Simple do/while loop to read from stream until
   // no bytes are returned
   do
   {
    // Read data (up to 1k) from the stream
    bytesRead = remoteStream.Read(buffer, 0, buffer.Length);

    // Write the data to the local file
    localStream.Write(buffer, 0, bytesRead);

    // Increment total bytes processed
    bytesProcessed += bytesRead;
   } while (bytesRead > 0);
  }
 }
}
catch (Exception e)
{
 Console.WriteLine(e.Message);
}
finally
{
 // Close the response and streams objects here 
 // to make sure they're closed even if an exception
 // is thrown at some point
 if (response != null) response.Close();
 if (remoteStream != null) remoteStream.Close();
 if (localStream != null) localStream.Close();
}

// Return total bytes processed to caller.
return bytesProcessed;
}

Answer 2

WebClient.OpenRead() might be what you're looking for.

Sample from the MSDN page linked above:

 Dim uriString as String
 uriString = "http://www.google.com"

 Dim myWebClient As New WebClient()

 Console.WriteLine("Accessing {0} ...", uriString)

 Dim myStream As Stream = myWebClient.OpenRead(uriString)

 Console.WriteLine(ControlChars.Cr + "Displaying Data :" + ControlChars.Cr)
 Dim sr As New StreamReader(myStream)
 Console.WriteLine(sr.ReadToEnd())

 myStream.Close()

Answer 3

The short answer, in C#, looks like

using(System.Net.WebClient client = new System.Net.WebClient())
{
  string html = client.DownloadString("http://www.google.com");
}