Character Encoding Problem

Question

Hi,

I need to save this onto database(mysql) and show it back. (my database is utf_general_ci)

I αм iиvisibłє łiкє αiя--- I αм αs iмρøяŧαиŧ αs øxygєи--- I αм łiviиg iи ŧЋє wøяłd øƒ мy dяєαмz I αм αłwαys ŧЋєяє ŧø Ћєłρ øŧЋєяz--- I αм busy buŧ иєvєя igиøяє αиy øиє I αм ŧЋє øиє wЋø cαяєz--- I łøvє ŧø sєє øŧЋєя łαugЋiиg I αм ŧЋє øиє wЋø bøяяøw øŧЋєяz søяяøw I αм ŧЋє øиє wЋøz иαugЋŧy buŧ иicє I αм łøsŧ iи мy ŧЋøugЋŧs--- I łøvє ŧø ŧαłк--- I łøvє ŧø sЋαяє--- I αм яєαdy ŧø gø αиy wЋєяє--- I łøvє ŧø ƒły buŧ døи’ŧ Ћαvє wiиgs— I wαиŧ ŧøø ŧøucЋ ŧЋє sкy łiмiŧs--- I αм єvił buŧ иøŧ dєvił--- I иєvєя ƒøłłøw αиy ŧяєиd--- I αм ƒuиłøviиg--- suм ŧiмє łøvє ŧø bє αłøиє--- I łøvє ŧø łivє---

However when the data is being saved and retrieved, this is what I get

��Ã�±LL à ¸£Ã��Ã��Ã��Ã��à ¸£â�¥â�� â�ï&iqu

Any help please? Is it something to change the collation of the database or some PHP functions that can handle this ?

Answer 1

I must point you toward Joel's article titled The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!). This article explains what character encodings are, why they're used, and how to manage them.

Your data obviously includes a lot of Unicode characters from a wide range, so you will have to learn about how both PHP and MySql handle character encodings (in your configuration) and what functions to call to make sure that the encodings are translated correctly at the correct time. The sample (incorrect) data you pasted indicates that there is definitely an encoding mismatch somewhere.

Answer 2

You need to use 'SET NAMES utf8' to ensure your server sends results back using the UTF-8 character set. Example:

$dbconn = mysql_connect("localhost", "user", "pass") or die(mysql_error());
           mysql_select_db('database', $dbconn) or die(mysql_error());
           mysql_query("SET NAMES 'utf8'");

Answer 3

Make sure that MySQL mysql_query("SET NAMES 'utf8'"); is run for data transfer and then also make sure that each of your pages sends a UTF-8 header.

header('Content-Type: text/html; charset=utf-8');

Also, read Am I correctly Supporting UTF-8 to double check all everything and run some other test data through it to make sure it works. And you can test for UTF-8 on each page load.

Last, make sure that the browser has the charset you want to use installed because some of the new ones like Chrome don't automatically use some charsets. However, they all use UTF-8 so make sure you send that header!

Answer 4

To use special chars without encoding problems try to follow this 4 rules:

1. create fields on db with utf8_general_ci collation
2. make the query mysql_query("SET NAMES 'utf8';"); after any connection (you need mysql 5)
3. put meta charset utf8 tag in the html head block
4. read and write strings to db and to html without using utf8_encode or other functions

This text comes from here.

Answer 5

Try the following steps:

1. Make sure all your DB fields are set up to store UTF-8 data (utf8_general_ci).
2. Upon connection to the DB execute the following query: SET NAMES 'utf8';
3. Save your .php file as UTF-8 without BOM, there is a nice free text editor for Windows that will allow you to easily change (on the status bar) the encoding of the file, just select UTF-8 Plain.

That should work, however if you're receiving the data from the user (via GET, POST or COOKIE arguments) you may want to follow some additional steps described in this answer.

Also, regarding the following meta tag:

<meta http-equiv="content-type" content="text/html; charset=utf-8" />

Or the equivalent PHP header:

header('Content-Type: text/html; charset=utf-8');

You only need those to display UTF-8 data, it doesn't interfere with the storing process, although it doesn't hurt to use it, of course. =)