My objective is to [semi]automatically assign texts to different categories. There's a set of user defined categories and a set of texts for each category. The ideal algorithm should be able to learn from a human-defined classification and then classify new texts automatically. Can anybody suggest such an algorithm and perhaps .NET library that implements ше?
+1
A:
Doing this is not trivial. Obviously you can build a dictionary that maps certain keywords to categories. Just finding a keyword would suggest a certain category.
Yet, in natural language text, the keywords would usually not be in their stem form. You would need some morphology tools to find the stem form and use it on the dictionary.
But then somebody could write something like: "This article is not about ...". This would introduce the need for syntax and semantical analysis.
And then you would find that certain keywords can be used in several categories: "band" could be used in musics, Technics, or even handicraft work. You would therefore need an ontology and statistical or other methods to weigh the probability of the category to choose if not definite.
Some of the keywords might not even be easy to fit into an ontology: is mathematician closer to programmer or gardener? But you said in your question that the categories are built by men, so they could also help building the ontology.
Have a look on computational linguistics here and in Wikipedia for further studies.
Now, the more narrow the field your texts are from, the more structured they are, and the smaller the vocabulary, the easier the problem becomes.
Again some keywords for further studies: morphology, syntax analysis, semantics, ontology, computational linguistics, indexing, keywording
Ralph Rickenbach
2010-08-27 13:28:39
+2
A:
There are multiple approaches to automatic text classification. A naive Bayes classifier is possibly the simplest of them. Another one is the K-nearest neighbor that you can use. This google answer on categorization of text might help you.
Gangadhar
2010-08-27 13:29:37
A:
The general term for these methods is "multivariate methods". That with a search on "text classification" or "text categorization" should bring up some useful leads. Good luck !
Grembo
2010-08-27 16:08:42
A:
Support vector machine. Everyone loves support vector machines. You'll need to do quite a bit of reading, and perhaps even buy a book. But you could start by reading a paper to see if you like the idea.
Tom Anderson
2010-08-27 16:59:54
A friend who knows a lot more about this than me says "An SVM would indeed be a smart choice Tom. There are more efficient techniques that will give you similar results if you have large datasets though... how many training samples per category?", to which i replied "Not sure, i ask for someone else. But not a lot, i think.", to which he in turn replied "OK well the simple answer is that an SVM would be a good place to start.". So now you know.
Tom Anderson
2010-08-30 21:46:20