Finding trending topics from a stream of data

Question

Finding single word trend is simple you can chunk each word of the data stream and do a count and limit it by last 24 hrs or 48 hrs. I'm not sure how to find trends of 2 word or 3 word combination? Any help is apprciated

Answer 1

So you've got something - for the single-word case - that says something along the lines of:

while (true)
    word = readNextWord()
    register(word, now)
    discardWordsOlderThan (now - windowSize)

Just keep track of the previous word:

while (true)
    word = readNextWord()
    register(prev + " " + word, now)
    prev = word
    discardWordsOlderThan (now - windowSize)