views:

497

answers:

2

I'm currently writing some tests so that I may improve my skills with the Internet interaction through Windows Forms. One of those tests is to find a postal code which should be returned by Canada Post website.

  1. My default URL setting is set to: http://www.canadapost.ca/cpotools/apps/fpc/personal/findByCity?execution=e4s1
  2. The required form fields are: streetNumber, streetName, city, province
  3. The contentType is "application/x-www-form-enclosed"

EDIT: Please consider the value "application/x-www-form-encoded" instead of point 3 value as the contentType. (Thanks EricLaw-MSFT!)

The result I get is not the result expected. I get the HTML source code of the page where I could manually enter the information to find the postal code, but not the HTML source code with the found postal code. Any idea of what I'm doing wrong?

Shall I consider going the XML way? Is it first of all possible to search on Canada Post anonymously?

Here's a code sample for better description:

public static string FindPostalCode(ICanadadianAddress address) {
   var postData = string.Concat(string.Format("&streetNumber={0}", address.StreetNumber)
    , string.Format("&streetName={0}", address.StreetName)
    , string.Format("&city={0}", address.City)
    , string.Format("&province={0}", address.Province));

   var encoding = new ASCIIEncoding();
   byte[] postDataBytes = encoding.GetBytes(postData);
   request = (HttpWebRequest)WebRequest.Create(DefaultUrlSettings);
   request.ImpersonationLevel = System.Security.Principal.TokenImpersonationLevel.Anonymous;
   request.Container = new CookieContainer();
   request.Timeout = 10000;
   request.ContentType = contentType;
   request.ContentLength = postDataBytes.LongLength;
   request.Method = @"post";
   var senderStream = new StreamWriter(request.GetRequestStream());
   senderStream.Write(postDataBytes, 0, postDataBytes.Length);
   senderStream.Close();
   string htmlResponse = new StreamReader(request.GetResponse().GetResponseStream()).ReadToEnd();

   return processedResult(htmlResponse); // Processing the HTML source code parsing, etc.
}

I seem stuck in a bottle neck in my point of view. I find no way out to the desired result.

EDIT: There seems to have to parameters as for the ContentType of this site. Let me explain.

  • There's one with the "meta"-variables which stipulates the following:

meta http-equiv="Content-Type" content="application/xhtml+xml, text/xml, text/html; charset=utf-8"

  • And another one later down the code that is read as:

form id="fpcByAdvancedSearch:fpcSearch" name="fpcByAdvancedSearch:fpcSearch" method="post" action="/cpotools/apps/fpc/personal/findByCity?execution=e1s1" enctype="application/x-www-form-urlencoded"

My question is the following: With which one do I have to stick?

Let me guess, the first ContentType is to be considered as the second is only for another request to a function or so when the data is posted?

EDIT: As per request, the closer to the solution I am is listed under this question: WebRequest: How to find a postal code using a WebRequest against this ContentType=”application/xhtml+xml, text/xml, text/html; charset=utf-8”?

Thanks for any help! :-)

+1  A: 

I'm trying to see a reason why you are not using the WebClient class:-

var fields = new NameValueCollection();
fields.Add("streetnumber", address.StreetNumber);
fields.Add("streetname", address.StreetName);
fields.Add("city", address.City);
fields.Add("province", address.Province);

var wc = new WebClient();
byte[] resultData = wc.UploadValues(url, fields);
string result = Encoding.Default.GetString(resultData);

You might want to check the encoding used by the server when sending the results, if it uses UTF-8 change the last line to:-

string result = Encoding.UTF8.GetString(resultData);

Some issues I spot in your orginal code:-

  1. The first field is prefixed with &, that shouldn't be there.
  2. You need call use Uri.EscapeDataString on each field value.
  3. You are attempting to construct a memory stream around the result of GetRequestStream, I can't see what that would achieve even if MemoryStream had such a constructor but it doesn't anyway. Just write directly to the stream returned by GetRequestStream

If you have already done so get yourself a copy of fiddler so you can observe what occurs when a standard form requests the data sucessfully and what your code is doing.

Edit: If you have evidence that the lack of a cookie container is what causes WebClient not to work then you could try this approach:-

public class MyWebClient : WebClient
{

    protected override WebRequest GetWebRequest (Uri address)
    {
      WebRequest request = (WebRequest) base.GetWebRequest (address);

      request.Container = new CookieContainer();
      return request;
    }
}

Now use my code above to but instead od instancing WebClient instance MyWebClient instead.

AnthonyWJones
The only reason is that I couldn't find a way to manage cookies automatically. With no cookies handler, the responses get redirected too many times and I get a WebException.
Will Marcouiller
Thanks for your precious guidelines! I shall try your code suggestion and give you feedback accordingly. As for the MemoryStream constructor, I have mistaken with a StreamWriter (shame on me! ;-))
Will Marcouiller
But what about the cookies handling?
Will Marcouiller
I brought the correction on the MemoryStream() constructor and changed it for a StreamWriter(). :-)
Will Marcouiller
@Will: Its still unnecessary, StreamWriter allows your write text to a stream but you already have your text encoded to a byte array. Just use the stream given by GetRequestStream() directly.
AnthonyWJones
Understood! =) I shall do it directly with the instance of Stream class returned by GetRequestStream(). Be sure I will let you know when I try.
Will Marcouiller
I tried the code you suggested me, and the result is that I still get the whole HTML source code of the querying page, and not the one that holds the resulting postal code. :-( Thus I have noticed an improved performance. :-) That said, I have noticed that the content-type seems to be XML or so. Then, this might be the cause why it doesn't do what I wish to. What would be the way to go with an XML content type?
Will Marcouiller
@Anthony: I clicked the wrong place and SOF allowed the change for the up-vote... =( Please edit or something so that I can up-vote again.
Will Marcouiller
@Will: You really need to use the site normally with a browser with fiddler in place. Then carefully examine the conversation and compare with the one generated by your code __something__ must vary, what is it?
AnthonyWJones
+1  A: 

HTTPWebRequest will return the content of the URLrequested. If it is a HTML page, it will return HTML markup. If the page content is XML, then it will return XML markup.

It sounds like you need is a Webservice. I would see if that site has any webserivces available to process that type of request. If they do, then it will return XML, JSON markup in response to your query. Otherwise you are left to parsing the output of the request.

Scott Lance
How can I find whether they have WebServices available except by inquiring directly to their IT services?
Will Marcouiller
And what is JSON again? I'm quite new to this stuff though am I an experienced programmer. Thus, I never really went the Web-way. It's a new beginning for me! :-)
Will Marcouiller
Look at their products and services. Or do a search for postal code webservice. I would look at: http://www.geonames.org/export/
Scott Lance
If I get the HTML markup as the response, it's okay for me as I only have to parse the file in order to get my answer, the postal code found. However, if there are better practices, I would like to learn the "good way".
Will Marcouiller
JSON is JavaScript Object Notation. See http://www.json.org/
Scott Lance
Thanks Scott for sharing such information freely. In less than 5 minutes, you gave me enough information to shut my mouth for a few days as I shall read and get acquainted with this stuff! Hehehe...
Will Marcouiller