regex to match any UTF character excluding punctuation

Question

Hello,

I'm preparing a function in PHP to automatically convert a string to be used as a filename in a URL (*.html). Although ASCII should be use to be on the safe side, for SEO needs I need to allow the filename to be in any language but I don't want it to include punctuation other than a dash (-) and underscore (_), chars like *%$#@"' shouldn't be allowed.

Spaces should be converted to dashes.

I think that using Regex will be the easiest way, but I'm not sure it how to handle UTF8 strings.

My ASCII functions looks like this:

function convertToPath($string)
{
    $string = strtolower(trim($string));
    $string = preg_replace('/[^a-z0-9-]/', '-', $string);
    $string = preg_replace('/-+/', "-", $string);
    return $string;
}

Thanks,

Roy.

Answer 1

If UTF-8 mode is selected you can select all non-Letters (according to the Unicode general category - please refer to the PHP documentation Regular Expression Details) by using

/\P{L}+/

so I'd try the following (untested):

function convertToPath($string)
{
    $string = mb_strtolower(trim($string), 'UTF-8');
    $string = preg_replace('/\P{L}+/', '-', $string);
    $string = preg_replace('/-+/', "-", $string);
    return $string;
}

Be aware that you'll get prolems with strtolower() on UTF-8 strings as it'll mess with you multi-byte characters - use mb_strtolower() instead.

Answer 2

I think that for SEO needs you should stick to ASCII characters in the URL.

In theory, many more characters are allowed in URLs. In practice most systems only parse ASCII reliable.

Also, many automagically-parse-the-link scripts choke on non-ASCII characters. So allowing URLs with non-ASCII characters in your URLs drastically reduces the change of your link showing up (correctly) in user generated content. (if you want an example of such a script, take a look at the stackoverflow script, it chokes on parenthesis for example)

You could also take a look at: How to handle diacritics (accents) when rewriting ‘pretty URLs’

The accepted solution there is to transiterate the non-ASCII characters:

<?php
  $text = iconv('UTF-8', 'US-ASCII//TRANSLIT', $text);
?>

Hope this helps