HttpWebRequest: How to find a postal code at Canada Post through a WebRequest with x-www-form-enclosed?

Question

I'm currently writing some tests so that I may improve my skills with the Internet interaction through Windows Forms. One of those tests is to find a postal code which should be returned by Canada Post website.

1. My default URL setting is set to: http://www.canadapost.ca/cpotools/apps/fpc/personal/findByCity?execution=e4s1
2. The required form fields are: streetNumber, streetName, city, province
3. The contentType is "application/x-www-form-enclosed"

EDIT: Please consider the value "application/x-www-form-encoded" instead of point 3 value as the contentType. (Thanks EricLaw-MSFT!)

The result I get is not the result expected. I get the HTML source code of the page where I could manually enter the information to find the postal code, but not the HTML source code with the found postal code. Any idea of what I'm doing wrong?

Shall I consider going the XML way? Is it first of all possible to search on Canada Post anonymously?

Here's a code sample for better description:

public static string FindPostalCode(ICanadadianAddress address) {
   var postData = string.Concat(string.Format("&streetNumber={0}", address.StreetNumber)
    , string.Format("&streetName={0}", address.StreetName)
    , string.Format("&city={0}", address.City)
    , string.Format("&province={0}", address.Province));

   var encoding = new ASCIIEncoding();
   byte[] postDataBytes = encoding.GetBytes(postData);
   request = (HttpWebRequest)WebRequest.Create(DefaultUrlSettings);
   request.ImpersonationLevel = System.Security.Principal.TokenImpersonationLevel.Anonymous;
   request.Container = new CookieContainer();
   request.Timeout = 10000;
   request.ContentType = contentType;
   request.ContentLength = postDataBytes.LongLength;
   request.Method = @"post";
   var senderStream = new StreamWriter(request.GetRequestStream());
   senderStream.Write(postDataBytes, 0, postDataBytes.Length);
   senderStream.Close();
   string htmlResponse = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();

   return processedResult(htmlResponse); // Processing the HTML source code parsing, etc.
}

I seem stuck in a bottle neck in my point of view. I find no way out to the desired result.

EDIT: There seems to have to parameters as for the ContentType of this site. Let me explain.

• There's one with the "meta"-variables which stipulates the following:

meta http-equiv="Content-Type" content="application/xhtml+xml, text/xml, text/html; charset=utf-8"

• And another one later down the code that is read as:

form id="fpcByAdvancedSearch:fpcSearch" name="fpcByAdvancedSearch:fpcSearch" method="post" action="/cpotools/apps/fpc/personal/findByCity?execution=e1s1" enctype="application/x-www-form-urlencoded"

My question is the following: With which one do I have to stick?

Let me guess, the first ContentType is to be considered as the second is only for another request to a function or so when the data is posted?

EDIT: As per request, the closer to the solution I am is listed under this question: WebRequest: How to find a postal code using a WebRequest against this ContentType=”application/xhtml+xml, text/xml, text/html; charset=utf-8”?

Thanks for any help! :-)

Answer 1

I'm trying to see a reason why you are not using the WebClient class:-

var fields = new NameValueCollection();
fields.Add("streetnumber", address.StreetNumber);
fields.Add("streetname", address.StreetName);
fields.Add("city", address.City);
fields.Add("province", address.Province);

var wc = new WebClient();
byte[] resultData = wc.UploadValues(url, fields);
string result = Encoding.Default.GetString(resultData);

You might want to check the encoding used by the server when sending the results, if it uses UTF-8 change the last line to:-

string result = Encoding.UTF8.GetString(resultData);

Some issues I spot in your orginal code:-

1. The first field is prefixed with &, that shouldn't be there.
2. You need call use Uri.EscapeDataString on each field value.
3. You are attempting to construct a memory stream around the result of GetRequestStream, I can't see what that would achieve even if MemoryStream had such a constructor but it doesn't anyway. Just write directly to the stream returned by GetRequestStream

If you have already done so get yourself a copy of fiddler so you can observe what occurs when a standard form requests the data sucessfully and what your code is doing.

Edit: If you have evidence that the lack of a cookie container is what causes WebClient not to work then you could try this approach:-

public class MyWebClient : WebClient
{

    protected override WebRequest GetWebRequest (Uri address)
    {
      WebRequest request = (WebRequest) base.GetWebRequest (address);

      request.Container = new CookieContainer();
      return request;
    }
}

Now use my code above to but instead od instancing WebClient instance MyWebClient instead.

Answer 2

HTTPWebRequest will return the content of the URLrequested. If it is a HTML page, it will return HTML markup. If the page content is XML, then it will return XML markup.

It sounds like you need is a Webservice. I would see if that site has any webserivces available to process that type of request. If they do, then it will return XML, JSON markup in response to your query. Otherwise you are left to parsing the output of the request.