Strange Encoding Problem

Question

I have a table of datas encoded in latin5 charset and all the columns in the table are also latin5. From mysql console when I enter "SET NAMES 'latin5'" and query the table results are ok . When I try to delete or insert/update all the new data's encodings are perfect. But when I try to insert Iso-8859 data (also verify this with mb_detect_encoding) to the database and I try to insert the data without "SET NAMES" it doesn't insert/update/select in proper encodings or when I used "SET NAMES 'latin5'" it doesn't insert/update in proper way but select are ok latin5 datas are coming in proper encodings in with only set names 'latin5'. When i use set names 'utf8' the select queries are bad encoded but insert/update are ok.

The reason I asked that we will go to production. And this makes me thinking about possible future problems.

Answer 1

mb_detect_encoding doesn't know what encoding your string is. It makes a qualified guess, but there are no guarantees that it will guess right. Especially not if the candidates are all single-byte encodings, as in the case of latin1 and latin5.

There really is no substitute for knowing what you're doing, if you want to get charsets right. I suggest that you read these pages at least a couple of times:

• http://www.phpwact.org/php/i18n/charsets
• http://www.nicknettleton.com/zine/php/php-utf-8-cheatsheet

In particular, make note that a web page is served with a http header, that specifies the charset that the page is encoded with. Unless you explicitly set this from your php-script, you'll use the webservers default, which may vary from server to server.

Also, be wary to actually understand what is going on, rather than doing trial and error. The latter can easily get you something that works in some context, but not in every context.

And lastly. If you have any choice at all, I seriously suggest that you use utf-8 for everything. latin5 is going to get you lots of grief.