can NLTK/pyNLTK work "per language" (i.e. non-english), and how?

Question

how can I tell nltk to treat the text in a particular language?

BKG: once in a while i write a specialized NLP routine to do POS tagging, tokenizing etc. on a non-english (but still hindo-european) text domain.

this question seem to address only different corpora, not the change in code / settings: http://stackoverflow.com/questions/1639855/nltk-tagging-in-german

alternatively, are there any specialized hebrew / spanish / polish NLP module for python?

thanks

Answer 1

I'm not sure what you're referring to as the changes in code/settings. NLTK mostly relies on machine learning and the "settings" are usually extracted from the training data.

When it comes to POS tagging the results and tagging will be dependant on the tagger you use/train. Should you train your own you'll of course need some spanish / polish training data. The reason these might be hard to find is the lack of gold standard material publicly available. There are tools out there to do that do this, but this one isn't for python (http://www.ims.uni-stuttgart.de/projekte/corplex/TreeTagger/).

The nltk.tokenize.punkt.PunktSentenceTokenizer tokenizer will tokenize sentences according to multilingual sentence boundaries the details of which can be found in this paper (http://www.mitpressjournals.org/doi/abs/10.1162/coli.2006.32.4.485).