How does HttpWebRequest differ (functional) from pasteing a URL into an address bar?

Question

I'm narrowing in on an underlying problem related to two prior questions.

Basically, I've got a URL that when I fetch it manually (paste it into browser) works just fine, but when I run through some code (using the HttpWebRequest) has a different result.

The URL (example):

http://208.106.250.207:8192/announce?info_hash=-%CA8%C1%C9rDb%ADL%ED%B4%2A%15i%80Z%B8%F%C&peer_id=01234567890123456789&port=6881&uploaded=0&downloaded=0&left=0&compact=0&no_peer_id=0&event=started

The code:

String uri = BuildURI(); //Returns the above URL
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(uri);
req.Proxy = new WebProxy();
WebResponse resp = req.GetResponse();
Stream stream = resp.GetResponseStream();
... Parse the result (which is an error message from the server claiming the url is incorrect) ...

So, how can I GET from a server given a URL? I'm obviously doing something wrong here, but can't tell what.

Either a fix for my code, or an alternative approach that actually works would be fine. I'm not wed at all to the HttpWebRequest method.

Answer 1

Well, the only they might differ is in the HTTP headers that get transmitted. In particular the User-Agent.

Also, why are you using a WebProxy? That is not really necessary and it most likely is not used by your browser.

The rest of your code is fine.. Just make sure you set up the HTTP headers correctly. Check this link out:

I would suggest that you get yourself a copy of WireShark and examine the communication that happens between your browser and the server that you are trying to access. Doing so will be rather trivial using WireShark and it will show you the exact HTTP message that is being sent from the browser.

Then take a look at the communication that goes on between your C# application and the server (again using WireShark) and then compare the two to find out what exactly is different.

If the communication is a pure HTTP GET method (i.e. there is no HTTP message body involved), and the URL is correct then the only two things I could think of are:

1. make sure that your are send the right protocol (i.e. HTTP/1.0 or HTTP/1.1 or whatever it is that you should be sending)
2. make sure that you are sending all required HTTP headers correctly, and obviously that you are not sending any HTTP headers that you shouldn't be sending.

Answer 2

There could be something wrong with the URL. Instead of using a string, it's usually better to use an instance of System.Uri:

String url = BuildURI(); //Returns the above URL
Uri uri = new Uri(url);
HttpWebRequest req = (HttpWebRequest)HttpWebRequest.Create(url);
req.Proxy = new WebProxy();
using (WebResponse resp = req.GetResponse()) {
    using (Stream stream = resp.GetResponseStream()) {
        // whatever
    }
}

Answer 3

I recommend you use Fiddler to trace both the "paste in web browser" call and the HttpWebRequest call.

Once traced you will be able to see any differences between them, whether they are differences in the request url, in the form headers, etc, etc.

It may actually be worth pasting the raw requests from both (obtained from Fiddler) here, if you can't see anything obvious.

Answer 4

I think you need to see exactly what's flowing to your server in the HTTP request. Does sound likely that the headers are interestingly different.

You can introduce a some kind of debugging proxy between your request and the server (for example RAD has such a capability in the box).