Encoding problem with HttpWebResponse

Answer 1

Firstly, the easier way of writing that code is to use a StreamReader and ReadToEnd:

HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(myURL);
using (HttpWebResponse response = (HttpWebResponse)webRequest.GetResponse())
{
    using (Stream resStream = response.GetResponseStream())
    {
        StreamReader reader = new StreamReader(resStream, Encoding.???);
        return reader.ReadToEnd();
    }
}

Then it's "just" a matter of finding the right encoding. How did you create the file? If it's with Notepad then you probably want Encoding.Default - but that's obviously not portable, as it's the default encoding for your PC.

In a well-run web server, the response will indicate the encoding in its headers. Having said that, response headers sometimes claim one thing and the HTML claims another, in some cases.

Answer 2

Jon Skeet got the answer at 50%. I would like to thank him a lot!

I had a problem reading (I posted the new code in the question) by using the Charset of the html file (Encoding was always empty) and I have to write the file with the Default of the user to display it back well.

Now it works well :)

Answer 3

Hi there DAok.

I'm having a very similar problem to you.

I just can't figure out the solution by reading this post.

I'm parsing google.com.ar and whenever i find a 'strange' character ,my program can't understand it properly.

I guess it's just an Encoding problem.

The thing is i can determine the proper encoding by using your code and i'm sending the right Encoding to the StreamReader constructor but still, when i write the results to a file all the weird characters are not there.

How did you solve it exactly?

This is my code:

        HttpWebRequest oRequest;
        HttpWebResponse oResponse;
        Encoding encoding;
        string googleUrl = "http://www.google.com.ar/search?q=";
        string pagedata;
        string charSet;

        oRequest = (HttpWebRequest)WebRequest.Create(googleUrl);

        oResponse = (HttpWebResponse)oRequest.GetResponse();

        oRequest.ContentType = "Text/HTML";

        charSet = oResponse.CharacterSet;

        if (String.IsNullOrEmpty(charSet))
             encoding = Encoding.Default;
         else
             encoding = Encoding.GetEncoding(charSet);

         StreamReader sr = new StreamReader(oResponse.GetResponseStream(), encoding);

        pagedata= sr.ReadToEnd();
        sr.Close();
        oResponse.Close();
        return pagedata;

Any pointer is appreciated!

Thanks a lot!

Answer 4

oResponse.CharacterSet is always = "ISO-8859-1" with every webpage. for example [http://www.google.com.vn/] have [charset=UTF-8] but oResponse.CharacterSet= "ISO-8859-1" [https://bazar.biglobe.ne.jp/cgi-bin/index.cgi] have [charset=euc-jp] but oResponse.CharacterSet= "ISO-8859-1" Do you have any another way to resovle?