How does Google News and Techmeme cluster news items that are similar? Are there any well know algorithm that is used to achieve this?
Appreciate your help.
Thanks in advance.
+2
A:
One fairly common way to cluster text based on content is to use Principle Component Analysis on the word vectors (a vector of n dimensions where each possible word represents one dimension and the magnitude in each direction, for each vector, is the number occurrences of the word in that particular article), followed by just a simple clustering such as K-Means.
Andy
2009-04-24 05:22:14
Thanks Andy. Appreciate your help. While researching this topic from your answer I found some useful links. I am posting it here as comment so that anyone interested in this topic can have a starting point. Hierarchical agglomerative clustering http://nlp.stanford.edu/IR-book/html/htmledition/hierarchical-agglomerative-clustering-1.htmlA Tutorial on Clustering Algorithmshttp://home.dei.polimi.it/matteucc/Clustering/tutorial_html/kmeans.htmlIntroduction to Information Retrievalhttp://nlp.stanford.edu/IR-book/html/htmledition/irbook.html
niraj
2009-04-29 15:24:29
+1
A:
The algorithmic basis is agglomerative clustering or something similar. But there are a number of heuristics on top of that. For example, the vector space is surely comprised of words and phrases (word n-grams). Limiting the search in a strict time period is also very important. And identifying names, and weighing more the title and the paragraph headings are also key parts.
On a tangentially related note. If you are interested in finding near-duplicate articles then there are a number of easier to implement approaches, such as the one described here
Costas Boulis
2009-05-05 09:16:53