NLTK and language detection

views:

answers:

NLTK and language detection

How do I detect what language a text is written in using NLTK?

The examples I've seen use nltk.detect, but when I've installed it on my mac, I cannot find this package.

Cheers

Nik

+1 A:

Have you come across the following code snippet?

english_vocab = set(w.lower() for w in nltk.corpus.words.words())
text_vocab = set(w.lower() for w in text if w.lower().isalpha())
unusual = text_vocab.difference(english_vocab)

from http://groups.google.com/group/nltk-users/browse_thread/thread/a5f52af2cbc4cfeb?pli=1&safe=active

Or the following demo file?

http://code.google.com/p/nltk/source/browse/trunk/nltk_contrib/nltk_contrib/misc/langid.py

William 2010-08-02 02:34:33

Nope, I hadn't. Thank you very much, that was VERY useful :-)

niklassaers 2010-08-03 09:59:00

PS, it still relied on nltk.detect, though. Any idea on how to install that on a Mac?

niklassaers 2010-08-03 09:59:35

I don't believe detect is a native module for nltk.Here's the code: http://docs.huihoo.com/nltk/0.9.5/api/nltk.detect-pysrc.htmlYou could probably download it and put it in your python library, which may be in:/Library/Python/2.x/site-packages/nltk...

William 2010-08-03 13:53:12