views:

157

answers:

7

English, of course, is a no-brainer for regex because that's what it was originally developed in/for:

Can regular expressions understand this character set?

French gets into some accented characters which I'm unsure how to match against - i.e. are è and e both considered word characters by regex?

Les expressions régulières peuvent comprendre ce jeu de caractères?

Japanese doesn't contain what I know as regex word characters to match against.

正規表現は、この文字を理解でき、設定?

+1  A: 

As far as I know, there isn't any specific pattern you can use i.e. [a-zA-Z] to match "è", but you can always match them in separately, i.e. [a-zA-Zè正]

Obviously that can make your regexp immense, but you can always control this by adding your strings into variables, and only passing the variables into the expressions.

Marcos Placona
A: 

it is not about the regular expression but about framework that executes it. java and .net i think are very good in handling unicode. so "è and e both considered word characters by regex" is true.

Andrey
+4  A: 

Short answer: yes.

More specifically it depends on your regex engine supporting unicode matches (as described here).

Such matches can complicate your regular expressions enormously, so I can recommend reading this unicode regex tutorial (also note that unicode implementations themselves can be quite a mess so you might also benefit from reading Joel Spolsky's article about the inner workings of character sets).

Lars Tackmann
Note that Unicode is not the mess. It's all the attempts that came before it that makes the entire matter messy.
Joachim Sauer
By definition in that article, Unicode can't be a mess: implementations can be.
Tom
A: 

It depends on the implementation and the character set. In general the answer is "Yes," but it may require additional setup on your part.

In Perl, for example, the meaning of things like \w is altered by the chosen locale (use locale).

Sorpigal
+1  A: 

Generally speaking, regex is more for grokking machine-readable text than for human-readable text. It is in many ways a more general answer to the whole XML with regex thing; regex is by its very nature incapable of properly parsing human language, because the language is more complex than what you are using to parse it.

If you want to break down human language (English included), you would want to use a language analysis tool or even an AI, not mere regular expressions.

Williham Totland
A: 

This SO thread might help. It includes the Unicode character classes you can use in a regex (e.g., [Ll] is all lowercase letters, regardless of language).

Tom
Use in a regex in what engine? Perl? Boost? Java?
David Thornley
6.2L V8. What other kind is there?
Tom
A: 

/[\p{Latin}]/ should for example, include Latin alphabet. You can get the full explanation and reference here.

henasraf
That's a useful-looking site, but it concentrates on Perl and similar regex engines. It isn't universal.
David Thornley
hmm yeah, I'm not sure what engine the asker uses, but maybe it's useful? Perl RegEx engine is used widely
henasraf