Say we have a UTF-8 string $s and we need to shorten it so it can be stored in N bytes. Blindly truncating it to N bytes could mess it up. But decoding it to find the character boundaries is a drag. Is there a tidy way?
[Edit 20100414] In addition to S.Mark’s answer: mb_strcut(), I recently found another function to do the job: grapheme_extract($s, $n, GRAPHEME_EXTR_MAXBYTES); from the intl extension. Since intl is an ICU wrapper, I have a lot of confidence in it.
No. There is no way to do this other than decoding. The coding is pretty mechanical however. See the pretty table in the wikipedia article
Edit: Michael Borgwardt shows us how to do it without decoding the whole string. Clever.
Edit: S.Mark's answer is actually better than mine - PHP has a (badly documented) builtin function that solves exactly this problem.
Original "back to the bits" answer follows:
This ensures that you don't have an incomplete character dangling at the end, which is the main thing that can go wrong when truncating UTF-8.
Unfortunately (as Andrew reminds me in the comments) there are also cases where two separately encoded Unicode code points form a single character (basically, diacritics such as accents can be represented as separate code point modifying the preceding letter).
Handling this kind of thing requires advanced Unicode-Fu which is not available in PHP and may not even be possible for all cases (there are somne weird scripts out there!), but fortunately it's relatively rare, at least for Latin-based languages.
I coded up this simple function for this purpose, you need mb_string though.
function str_truncate($string, $bytes = null)
{
if (isset($bytes) === true)
{
// to speed things up
$string = mb_substr($string, 0, $bytes, 'UTF-8');
while (strlen($string) > $bytes)
{
$string = mb_substr($string, 0, -1, 'UTF-8');
}
}
return $string;
}
While this code also works, S.Mark answer is obviously the way to go.
I think you don't need to reinvent the wheel, you could just use mb_strcut and make sure you set encoding to UTF-8 first.
mb_internal_encoding('UTF-8');
echo mb_strcut("\xc2\x80\xc2\x80", 0, 3); //from index 0, cut 3 characters.
its return
\xc2\x80
because in \xc2\x80\xc2, last one is invalid
Here's a test for mb_strcut(). It doesn't prove that it does just what we're looking for but I find it pretty convincing.
<?php
ini_set('default_charset', 'UTF-8' );
$strs = array(
'Iñtërnâtiônàlizætiøn',
'החמאס: רוצים להשלים את עסקת שליט במהירות האפשרית',
'ايران لا ترى تغييرا في الموقف الأمريكي',
'独・米で死傷者を出した銃の乱射事件',
'國會預算處公布驚人的赤字數據後',
'이며 세계 경제 회복에 걸림돌이 되고 있다',
'В дагестанском лесном массиве южнее села Какашура',
'นายประสิทธิ์ รุ่งสะอาด ปลัดเทศบาล รักษาการแทนนายกเทศมนตรี ต.ท่าทองใหม่',
'ભારતીય ટીમનો સુવર્ણ યુગ : કિવીઝમાં પણ કમાલ',
'ཁམས་དཀར་མཛེས་ས་ཁུལ་དུ་རྒྱ་གཞུང་ལ་ཞི་བའི་ངོ་རྒོལ་',
'Χιόνια, βροχές και θυελλώδεις άνεμοι συνθέτουν το',
'Հայաստանում սկսվել է դատական համակարգի ձեւավորումը',
'რუსეთი ასევე გეგმავს სამხედრო');
for ( $i = 10; $i <= 30; $i += 5 ) {
foreach ($strs as $s) {
$t = mb_strcut($s, 0, $i, 'UTF-8');
print(
sprintf('%3s%3s ', mb_strlen($t, 'UTF-8'), mb_strlen($t, 'latin1'))
. ( mb_check_encoding($t, 'UTF-8') ? ' OK ' : ' Bad ' )
. $t . "\n");
}
}
?>
In addition to S.Mark’s answer which was mb_strcut(), I recently found another function to do a similar job: grapheme_extract($s, $n, GRAPHEME_EXTR_MAXBYTES); from the intl extension.
The functionality is a bit different: mb_strcut() documentation claims it cuts at the nearest UTF-8 character boundary, so it doesn't respect multi-character graphemes while grapheme_extract(), otoh, does. So depending what you need, grapheme_extract() might be better (e.g. to display a string) or mb_strcut() might be better (e.g. for indexing). Anyway, just though I'd mention it.
(And since intl is an ICU wrapper, I have a lot of confidence in it.)