tags:

views:

165

answers:

2
+3  Q: 

Document similarity

Hi All, I used tf/idf to calculate consine similarity between two documents. It has some limitation and does not perform very well.

I looked for LDA (latent dirichlet allocation) to calculate document similarity. I don't know much about this. I couldn't find much stuff too about my problem.

Can you please provide me any tutorial related to my problem? Or can you give some advices how can i achive this task with LDA???

Thanks

P.S: also is there any source code availabe to perform such task with LDA??

A: 

You might be thinking of LSA (Latent Semantic Analysis) which is a very common solution to this kind of problem.

Pace  2010-02-17 01:57:13
Hi pace,Thanks for your reply. Yes i know about LSA and I also implemented it. I used JAMA package for SVD but I had a problem that if my rows are less than columsn it doesn't work :(. Can you tell me any other SMALL svd package?
 2010-02-17 02:08:04
+1  A: 

Have you had a look at Lucene and Mahout?

This might be useful - Latent Dirichlet Allocation with Lucene and Mahout.

Binary Nerd  2010-02-17 01:57:37
Thanks, can you also please answer is it possible to calculate similarity between two documents with the help of LDA? As mostly people said it can be used for un-supervised clustering :(
 2010-02-17 02:06:56
Sorry, i don't know enough about LDA to offer an experts answer to that, its not part of Mahout that i've used. However, my understanding of clustering is that your grouping objects based on some similarity measure, which in this case would be LDA.
Binary Nerd  2010-02-17 02:50:28

related questions

What are some good rigid body dynamics references?
Algorithm to find a common multiplier to convert decimal numbers to whole numbers
How to test randomness (case in point - Shuffling)
What are the core mathematical concepts a good developer should know?
How to get the base 10 logarithm of a Fixnum in Ruby?
Can you be terrible at math and still a great programmer?
Modulus operation with negatives values - weird thing ??
How do I calculate a trendline for a graph?
Calculate Video Duration
Russell's Paradox
Popularity algorithm
How do I calculate distance between two latitude longitude points?
The Google Calculator Glitch, could float vs. double be a possible reason?
Calculating Distance Between 2 Cities
Prime factors
Where can I find a "Math topic dependency tree" to assist my self-guided refresher on the subject?
How to Round Up The Result Of Integer Division
Do you have to be good at math to be a good programmer?
Useful math for programmers
Puzzle: Find largest rectangle (maximal rectangle problem)
What does the term "BODMAS" mean?
Mapping latitude/longitude to a distorted map
A little diversion into floating point (im)precision, part 1
Solving a linear equation
What's the difference between Math.Floor() and Math.Truncate() in .NET?