HTTP Post in C# console app doesn't return the same thing as a browser request

Question

I have a C# console app (.NET 2.0 framework) that does an HTTP post using the following code:

StringBuilder postData = new StringBuilder(100);
postData.Append("post.php?");
postData.Append("Key1=");
postData.Append(val1);
postData.Append("&Key2=");
postData.Append(val2);

byte[] dataArray = Encoding.UTF8.GetBytes(postData.ToString());

HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create("http://example.com/");
httpRequest.Method = "POST";
httpRequest.ContentType = "application/x-www-form-urlencoded";

httpRequest.ContentLength = dataArray.Length;
Stream requestStream = httpRequest.GetRequestStream();
requestStream.Write(dataArray, 0, dataArray.Length);
requestStream.Flush();
requestStream.Close();

HttpWebResponse webResponse = (HttpWebResponse)httpRequest.GetResponse();

if (httpRequest.HaveResponse == true) {
  Stream responseStream = webResponse.GetResponseStream();
  StreamReader responseReader = new System.IO.StreamReader(responseStream, Encoding.UTF8);
  String responseString = responseReader.ReadToEnd();
}

The outputs from this are:
webResponse.ContentLength = -1
webResponse.ContentType = text/html
webResponse.ContentEncoding is blank

The responseString is HTML with a title and body.

However, if I post the same URL into a browser (http://example.com/post.php?Key1=some_value&Key2=some_other_value), I get a small XML snippet like:

<?xml version="1.0" ?>
<RESPONSE RESULT="SUCCESS"/>

with none of the same HTML as in the application. Why are the responses so different? I need to parse the returned result which I am not getting in the HTML. Do I have to change how I do the post in the application? I don't have control over the server side code that accepts the post.

Answer 1

I believe the problem has something to do with the way your headers are set up for the WebRequest.

I have seen strange cases where attempting to simulate a browser by changing headers in the request makes a difference to the server.

The short answer is that your console application is not a web browser and the web server of example.com is expecting to interact with a browser.

You might also consider changing the ContentType to be "multipart/form-data".

What I find odd is that you are essentially posting nothing. The work is being done by the query string. Therefore, you probably should be using a GET instead of a POST.

Answer 2

In your code you a specify the post method, where when you POST the address in the browser it is most likely using a GET. Create a simple html form and specify POST as the method and your url as the action.

Answer 3

I've seen this in the past.

When you run from a browser, the "User-Agent" in the header is "Mozilla ...".

When you run from a program, it's different and generally specific to the language used.

Answer 4

I think you need to use a GET request, instead of POST. If the url you're using has querystring values (like ?Key1=some_value&Key2=some_other_value) then it's expecting a GET. Instead of adding post values to your webrequest, just put this data in the querystring.

HttpWebRequest httpRequest = (HttpWebRequest)WebRequest.Create("http://example.com/?val1=" + val1 + "&val2=" + val2);
httpRequest.Method = "GET";
httpRequest.ContentType = "application/x-www-form-urlencoded";
....

So, the result you're getting is different when you POST the data from your app because the server-side code has a different output when it can't read the data it's expecting in the querystring.

Answer 5

If you are indeed supposed to use the POST HTTP method, you have a couple things wrong. First, this line:

postData.Append("post.php?");

is incorrect. You want to post to post.php, you don't want post the value "post.php?" to the page. Just remove this line entirely.

This piece:

... WebRequest.Create("http://example.com/");

needs post.php added to it, so...

... WebRequest.Create("http://example.com/post.php");

Again this is assuming you are actually supposed to be POSTing to the specified page instead of GETing. If you are supposed to be using GET, then the other answers already supplied apply.

Answer 6

Is the form expecting a cookie? That is another possible reason why it works in the browser and not from the console app.

Answer 7

You'll want to get an HTTP sniffer tool like Fiddler and compare the headers that are being sent from your app to the ones being sent by the browser. There will be something different that is causing the server to return a different response. When you tweak your app to send the same thing browser is sending you should get the same response. (It could be user-agent, cookies, anything, but something is surely different.)

Answer 8

I need to capture the internet explorer activity any code help. I am using SHDocVw and mshtml dll, beforenavigate event but now capturing all the requests send on the web my email id [email protected]

Thanks in advance