How to ensure user submit only english text

Question

I am building a project involving natural language processing, since the nlp module currently only deal with english text, so I have to make sure the user submitted content (not long, only several words) is in english. Are there established ways to achieve this? Python or Javascript way preferred.

Answer 1

You could break the phrase up into words and check a dictionary (there are some that you can download, this may be of interest), but that would require that the dictionary you used was good enough.

It would also fall over for proper nouns (my name isn't in the dictionary for example).

Answer 2

If the content is long enough I would suggest some frequency analysis on the letters.

But for a few words I think your best bet is to compare them to an English dictionary and accept the input if half of them match.

Answer 3

Try:

http://wordlist.sourceforge.net/

For a list of English words.

You will need to be careful of names, e.g. "Canberra" or "Bill Clinton". These won't appear in the word list. I suggest just checking whether the first letter is capitalized as a first attempt.

Answer 4

I think the most effective way would be to ask the users to submit english text only :)

You can show a language selection drop-down over your text area with English/ Other as the options. When user selects "Other", disable the text area with a message that only English language is supported [at the moment].

Answer 5

Check the Language Recognition Chart

Answer 6

Try n-gram based statistical language recognition. This is a link to a demo of an algorithm using this technique, there is also a link to a paper describing the algorithm there. Try the demo, it performs quite well even on very short texts (3-4 words).

Answer 7

You are already doing NLP, if your module doesn't understand what language the text was then either the module doesn't work or the input was not in the correct language.

Answer 8

The Dictionary Switcher Firefox extensions has an option to detect the right dictionary as I type.
I guess it checks words against the installed dictionaries, and selects the one giving the less errors...

You can't expect all words of the text to be in the dictionary: abbreviations, proper nouns, typos... Beside, some words are common to several languages: a French rock group even made the titles of their disks to have a (different) meaning both in French and in English. So it is a statistical thing: if more than x% of words are found in a good English dictionary, chances are the user types in this language (even if there are mistakes, like probably in this answer, since I am not native English).

Answer 9

Google has a javascript API that has an implementation of language detection. I've only play tested with it, never used it in production.

http://code.google.com/apis/ajaxlanguage/documentation/#Detect

Answer 10

Maybe "Ensuring that the user submits only English text [PHP]" article will help you. The code is written in PHP, but is small enough to be easily rewritten.