translate by replacing words inside existing text

Question

What are common approaches for translating certain words (or expressions) inside a given text, when the text must be reconstructed (with punctuations and everythin.) ?

The translation comes from a lookup table, and covers words, collocations, and emoticons like L33t, CUL8R, :-), etc.

Simple string search-and-replace is not enough since it can replace part of longer words (cat > dog ≠> caterpillar > dogerpillar).

Assume the following input:

s = "dogbert, started a dilbert dilbertion proces cat-bert :-)"

after translation, i should receive something like:

result = "anna, started a george dilbertion process cat-bert smiley"

I can't simply tokenize, since i loose punctuations and word positions.

Regular expressions, works for normal words, but don't catch special expressions like the smiley :-) but it does .

re.sub(r'\bword\b','translation',s) ==> translation
re.sub(r'\b:-\)\b','smiley',s) ==> :-)

for now i'm using the above mentioned regex, and simple replace for the non-alphanumeric words, but it's far from being bulletproof.

(p.s. i'm using python)

Answer 1

The reason your smiley example doesn't work with regex is the \b refers to a word boundary. Since there are no "word" characters in the smiley, there is no word boundary, so your expression doesn't match. You could use lookaheads/lookbehinds to see if you are bounded by spaces, but to check against punctuation could be difficult considering your smileys are made of punctuation.

Answer 2

The problem is not that regexp can't match smileys (which is simply not true :P), but rather how your regular expression for that smiley is made.

The word boundary \b is described as follows in the python documentation:

Matches the empty string, but only at the beginning or end of a word. A word is defined as a sequence of Unicode alphanumeric or underscore characters, so the end of a word is indicated by whitespace or a non-alphanumeric, non-underscore Unicode character. Note that formally, \b is defined as the boundary between a \w and a \W character (or vice versa).

The problem now is that symbols like :, - and ) are itself word-boundaries, so they are especially not words, and as such won't match with \w. As such the space before the smiley is not recognized as a word boundary (simply because no word is following).

So if you want to match smileys you can't use \b but have to check for white spaces or something instead.

Answer 3

If you're looking for a non-regex solution, then here is my idea. Here are the steps I would use.

Preparation:

• Create a dictionary linking the words to be replaced to their replacements.
• Create a ternary tree of the words to be replaced.

Searching and replacing:

1. Split up the words by the spaces using split(). I use the term word to refer to a group of letters that doesn't contain a space.
2. Iterate through all of the words
   1. Search for the word in the ternary tree - if a partial match is found, check that the rest of the word is punctuation (or at least not stuff that would make it not be a match).
   2. Replace the word using the dictionary look-up if it was found in the ternary tree

You can read about ternary search trees here. There are ternary search tree python implementations, but you can make your own pretty simply. The main problem with this approach is if there is punctuation before the word (like a "), but that can be dealt with easily.