text mining library or lingual library ?

Question

i have a bunch of data harvested from a forum I own, and would like to do some text mining or use some linguistic library to extract useful information.

any text mining, data mining library in any language will do.

Thank you.

Answer 1

Mallet is a java library designed for text mining. Once you have preprocessed the text data, a general data mining tool like Weka would also suffice your task.

If you have access to SPSS or SAS, their products should be more easier to use.

Answer 2

You may like to have a look at the Python NLTK (Natural Language ToolKit): it's specifically designed for this kind of thing.

There is also a great book you can but to get you started.

Answer 3

I recommend that you have a look at R. It has an extensive number of text mining packages: have a look at the Natural Language Processing view. In particular, look at the tm package. Here are some relevant links:

• Paper about the package in the Journal of Statistical Computing: http://www.jstatsoft.org/v25/i05/paper. The paper includes a nice example of an analysis of the R-devel mailing list (https://stat.ethz.ch/pipermail/r-devel/) newsgroup postings from 2006.
• Package homepage: http://cran.r-project.org/web/packages/tm/index.html
• Look at the introductory vignette: http://cran.r-project.org/web/packages/tm/vignettes/tm.pdf

Another example of useful package for this is Gary King's readme package.