CakePHP is truncating a text field, probably encoding related

Question

Here's what I'm trying to do:

I'm parsing incoming email, and using it to create posts in the system. This works almost completely, but there's a few bugs to work out. The one that's currently giving fits is coming up when an email contains certain characters (for example, ® – “ ”), the email body is being truncated at the special character when inserting in to the database. It's happening during the $Model->save() call.

The email is not UTF-8 encoded (and since any user can send email, we can't control the incoming mail encoding), but the database is UTF-8. It seems to work correctly when the email comes in as UTF-8.

Any suggestions? Is there a way to convert the encoding before attempting to save?

Answer 1

My guess is that the database chokes at the point where you are trying to insert ISO-8859-1 (or some other encoding) special characters into a UTF-8 encoded mySQL table.

The only proper way to deal with this would be to determine the incoming E-Mail's encoding, and do an iconv() on it.

I've never dealt with incoming E-Mail in this detail but to determine UTF-8 encoding, I think this goes into the right direction: iconv_mime_decode

You could also try to sniff the encoding using mb_detect_encoding but this is deemed unreliable as far as I know.