Character encoding problem - PHP output, read by .NET, via HttpWebRequest

Question

I have a PHP script (running on a Linux server) that ouputs the names of some files on the server. It outputs these file names in a simple text-only format.

This output is read from a VB.NET program by using HttpWebRequest, HttpWebResponse, and a StreamReader.

The problem is that some of the file names being output contain... unusual characters. Specifically, the "section" symbol (§).

If I view the output of the PHP script in a web browser, the symbol appears fine.

But when I read the output of the PHP script into my .NET program, the symbol doesn't appear correctly (it appears as a generic "block" symbol).

I've tried all the different character encoding options that you can use when reading the response stream (from the HttpWebResponse). I've tried outputting the stream directly to a text file (no good), displaying it in a TextBox (no good), and even when viewing the results directly in the Visual Studio debugger, the character appears as a block instead of as the "section" symbol.

I've examined the output in a hex editor (as suggested by a related question, "how do you troubleshoot character encoding problems."

When I write out the section symbol (§) from .NET itself, the hex bytes I see representing it are "c2 a7" (makes sense if it's unicode, right? requires two bytes?). When I write out the output from the PHP script directly to a file and examine that with a hex editor, the symbol shows up as "ef bf bd" - three bytes instead of two?

I'm at a loss as to what to do - if I need to specify some other character encoding, or if I'm missing something obvious about this.

Here's the code that's used to get the output of the PHP script (VB-style comments modified so they appear correctly on this site):

Dim myRequest As HttpWebRequest = WebRequest.Create("http://www.example.com/sample.php")

Dim myResponse As HttpWebResponse = myRequest.GetResponse()

// read the response stream
Dim myReader As New StreamReader(myResponse.GetResponseStream())

// read the entire output in one block (just as an example)
Dim theOutput as String = myReader.ReadToEnd()

Any ideas?

• Am I using the wrong kind of StreamReader? (I've tried passing the character encoding in the call to create the new StreamReader - I've tried all the ones that are in System.Text.Encoding - UTF-8, UTF-7, ASCII, UTF-32, Unicode, etc.)
• Should I be using a different method for reading the output of the PHP script?
• Is there something I should be doing different on the PHP-side when outputting the text?

UPDATED INFO:

• The output from PHP is specifically encoded UTF-8 by calling: utf8_encode($file);
• When I wrote out the symbol from .NET, I copied and pasted the symbol from the Character Map app in Windows. I also copied & pasted it directly from the file's name (in Windows) and from this web page itself - all gave the same hex value when written out (c2 a7).
• Yes, the "section symbol" I'm talking about is U+00A7 (ALT+0167 on Windows, according to Character Map).
• The content-type is set explicitly via header('Content-Type: text/html; charset=utf-8'); right at the beginning of the PHP script.

UPDATE:

Figured it out myself, but I couldn't have done it without the help from the people who answered. Thank you!

Answer 1

Does PHP give you control over the encoding at all? It's generally not a good idea to just guess at it.

When you say you've written out the symbol from .NET, what encoding were you using? What actual Unicode code point is it? There's a section symbol at unicode U+00A7 - is that the one you mean? I've no idea why PHP would represent that as "ef bf bd" though.

Using a StreamReader should be fine, but you'll need to know the correct encoding.

EDIT: Okay, so it's meant to be UTF-8, and certainly isn't - so the problem is on the PHP side. If you run utf8_encode($file) and then print out the bytes of the result explicitly (without the web server getting in the way) what happens? I'm really surprised that a browser is managing to get the right symbol though... is this just plain HTML? Are you sure that all of "ef bf bd" is just the section symbol?

Is this web server public anywhere? If I could point my browser at it, I might be able to work out what's going on.

Answer 2

You're using utf8_encode($file), fine, but is PHP returning the content type as UTF-8 as well? Can you check the Content-Type header returned by your PHP page? You should particularly look at the charset field to make sure you have something like this:

Content-Type: text/html; charset=utf-8

I can see how a browser could be displaying the character correctly while .NET (rightfully or wrongfully) fails. Browsers usually try to be as robust and forgiving as possible. The browser you're using might be inferring the actual character encoding from the character sequences.

Answer 3

Figured it out!!

Like so many things, it's simple in retrospect!

Jon Skeet was correct - it was meant to be UTF-8, but definitely wasn't.

Turns out, in the original script I was using (before I stripped it down to make it simpler to debug), there was some additional text output by the script which was not wrapped in a utf8_encode() call. This caused the entire page to be output in ISO-8859-1 instead of UTF-8.

I noticed this when I checked my testing script's "encoding" property (in Firefox, "View Page Info"). It was UTF-8 for the testing script, but ISO-8859-1. The production script also printed the date of the file; this was not wrapped in a call to utf8_encode - and that caused the entire output to change to ISO-08859-1.

[Insert sound of me slapping my forehead here]

Thanks to everyone who answered! You were very helpful!