c# simple HttpWebRequest question

Question

The following code fails with a 400 bad request exception. My network connection is good and I can go to the site but I cannot get this uri with HttpWebRequest.

Any help is appreciated!

        private void button3_Click(object sender, EventArgs e)
        {
            WebRequest req = HttpWebRequest.Create(@"http://www.youtube.com/");
            try
            {
                //returns a 400 bad request... Any ideas???
                WebResponse response = req.GetResponse();
            }
            catch (WebException ex)
            {
                Log(ex.Message);                
            }
        }

Answer 1

Maybe you've got a proxy server running, and you haven't set the Proxy property of the HttpWebRequest?

Answer 2

There could be many causes for this problem. Do you have any more details about the WebException?

One cause, which I've run into before, is that you have a bad user agent string. Some websites (google for instance) check that requests are coming from known user agents to prevent automated bots from hitting their pages.

In fact, you may want to check that the user agreement for YouTube does not preclude you from doing what you're doing. If it does, then what you're doing may be better accomplished by going through approved channels such as web services.

Answer 3

First, cast the WebRequest to an HttpWebRequest like this:

HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(@"http://www.youtube.com/");

Then, add this line of code:

req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";

Answer 4

Set UserAgent and Referer in your HttpWebRequest:

var request = (HttpWebRequest)WebRequest.Create(@"http://www.youtube.com/");
request.Referer = "http://www.youtube.com/"; // optional
request.UserAgent =
    "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; WOW64; " +
    "Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; " +
    ".NET CLR 3.5.21022; .NET CLR 3.5.30729; .NET CLR 3.0.30618; " +
    "InfoPath.2; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)";
try
{
    var response = (HttpWebResponse)request.GetResponse();
    using (var reader = new StreamReader(response.GetResponseStream()))
    {
        var html = reader.ReadToEnd();
    }
}
catch (WebException ex)
{
    Log(ex);
}