Searching unicode text using regex

Question

Searching a file which is written in Hindi(Devanagri) (UTF-16) gave rise to the following problem.

The file contains:

त्रास ततत जुग नींद ना हा बु

Note that the first char 'त्र' is a multiple code point of त + ् + र Now while searching for 'त' I get 4 matches including the त of the first char. I am using Java.

How can I go about searching for 'त''s which are not part of multiple code point chars.

Any help will be appreciated. :)

Answer 1

You can do this using unicode properties, I believe.

त(?!\p{M}+)

Should match the त code point as long as it is not followed by any code points in the M category, which are characters intended to be combined with other characters. It uses a negative lookahead to make that assertion.

E: and if that doesn't work right away, try

\uxxxx(?!\p{M}+)

Where the xxxx is the number of the त symbol's code point.

Answer 2

It seems that the glyph 'त्र' is actually a ligature or conjunct, not a multiple code point character. So I guess you get the expected result (unless you want to match glyphs). See http://en.wikipedia.org/wiki/Devanagari#Conjuncts.