Access the Contents of a Web Page with C#

Question

I am trying to use C# to access the content of a webpage. For example, I want to grab the text of the body of google homepage.

I know this is doable in C# with its web browser control. But I couldn't find a good, simple example of doing it. All the resources I found online involve creating Forms and GUI, which I don't need, I just need a good old Console Application.

If anyone can provide a simple console-based code snippet that accomplishes the above, it'll be greatly appreciated.

Answer 1

Actually the WebBrowser is a GUI control used in case you want to visualize a web page (embed and manage Internet Explorer in your windows application). If you just need to get the contents of a web page you could use the WebClient class:

class Program
{
    static void Main(string[] args)
    {
        using (var client = new WebClient())
        {
            var contents = client.DownloadString("http://www.google.com");
            Console.WriteLine(contents);
        }
    }
}

Answer 2

If you just want the content and not an actual browser, you can use an HttpWebRequest.

Here's a code sample: http://www.c-sharpcorner.com/Forums/ShowMessages.aspx?ThreadID=58261

Answer 3

The HTML Agility Pack might be what you need. It provides access to HTML pages via DOM and XPath.

Answer 4

You can do something like this:

Uri u = new Uri( @"http://launcher.worldofwarcraft.com/alert" );
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(u);
HttpWebResponse res = (HttpWebResponse)req.GetResponse();
System.IO.Stream st = res.GetResponseStream();
System.IO.StreamReader sr = new System.IO.StreamReader(st);
string body = sr.ReadToEnd();
System.Console.WriteLine( "{0}", body );

the above code shows the maintenance message for WoW USA (if any message has been posted)

Answer 5

You can also use the WatiN library to load and manipulate web pages easily. This was designed as a testing library for web UI's. To use it get the latest from the official site http://watin.sourceforge.net/ . For C# the following code in a console application will give you the HTML of the Google home page (this is modified from the getting started example on the WatiN site). The library also contains many more useful methods for getting and setting various parts of the page, taking actions and checking for results.

   using System;
    using WatiN.Core;

    namespace Test
    {
      class WatiNConsoleExample
      {
        [STAThread]
        static void Main(string[] args)
        {
          // Open an new Internet Explorer Window and
          // goto the google website.
          IE ie = new IE("http://www.google.com");

          // Write out the HTML text of the body
          Console.WriteLine(ie.Text);


          // Close Internet Explorer and the console window immediately.
          ie.Close();

          Console.Readkey();
        }
      }
    }

Answer 6

Google screen scraping and as mentioned above use the HttpWebRequest. When you do whatever it is you're doing, I'd recommend using Fiddler to help you figure out what's really going on.