How to use regex for utf8 in ruby

Question

In RoR,how to validate a Chinese or a Japanese word for a posting form with utf8 code.

In GBK code, it uses [\u4e00-\u9fa5]+ to validate Chinese words. In Php, it uses /^[\x{4e00}-\x{9fa5}]+$/u for utf-8 pages.

Answer 1

Ruby 1.8 has poor support for UTF-8 strings. You need to write the bytes individually in the regular expression, rather then the full code:

>> "acentuação".scan(/\xC3\xA7/)
=> ["ç"]

To match the range you specified the expression will become a bit complicated:

/([\x4E-\x9E][\x00-\xFF])|(\x9F[\x00-\xA5])/  # (untested)

That will be improved in Ruby 1.9, though.

Edit: As noted in the comments, the unicode characters \u4E00-\u9FA5 only map to the expression above in the UTF16-BE encoding. The UTF8 encoding is likely different. So you need to analyze the mapping carefully and see if you can come up with a byte-matching expression for Ruby 1.8.

Answer 2

activeSupport has a UTF-8 handler

http://api.rubyonrails.org/classes/ActiveSupport/Multibyte/Handlers/UTF8Handler.html

---

otherwise, look in ruby 1.9, encoding method for Regexp objects

Answer 3

The Oniguruma regexp engine has proper support for Unicode. Ruby 1.9 uses Oniguruma by default. Ruby 1.8 can be recompiled to use it.

With Oniguruma you can use the exact same regex as in PHP, including the /u modifier to force Ruby to treat the string as UTF-8.

Answer 4

This is what i have done:

%r{^[#{"\344\270\200"}-#{"\351\277\277"}]+$}

This is basically a regular expression with the octal values that represent the range between U+4E00 and U+9FFF, the most common Chinese and Japanese characters.