Hello, I have a database where users upload articles. I would like to make an algorithm where my web app will suggest similar texts according to the one the user reads.
I saw some examples like Levenshtein distance. But those algorithms measures distance for strings and not for whole articles. Is there a way to extract most significant keywords from text? Surely, I understand that "most significant" is an ambiguous term.
How do other sites manage this?
thanks a lot
Is there a way to extract most significant keywords from text?
Yes. Basically, you extract all the words from the text, sort the words by frequency, eliminate the common words (a, an, the, etc.) by matching them against a common word dictionary, and save the top 20 or more words, along with their frequency, from each article.
The number of top words you save is related to both the length of the article and the subject matter of all the articles. Less words work for general interest articles, while more words are necessary for special interest articles, like answers to programming questions.
Articles that match more than half of the top words could be considered related. The degree of relatedness would depend on the number of matching top words and the frequencies of the matching words.
You could calculate a relatedness score by multiplying the frequencies of each matched word from the two articles and summing all the products. The higher the score, the more the articles are related.