Feed aggregator using hbase. How to design the schema?

Question

I am working on a project involving monitoring a large number of rss/atom feeds. I want to use hbase for data storage and I have some problems designing the schema. For the first iteration I want to be able to generate an aggregated feed (last 100 posts from all feeds in reverse chronological order).

Currently I am using two tables:

Feeds: column families Content and Meta : raw feed stored in Content:raw
Urls: column families Content and Meta : raw post version store in Content:raw and the rest of the data found in RSS stored in Meta

I need some sort of index table for the aggregated feed. How should I build that? Is hbase a good choice for this kind of application?

Question update: Is it possible( in hbase) to design a schema that could efficiently answer to queries like the one listed bellow?

SELECT data FROM Urls ORDER BY date DESC LIMIT 100

Answer 1

Peter Rietzler answer on hbase-user mail list:

Hi

In our project we are handling event lists where we have similar requirements. We do ordering by choosing our row keys wisely. We use the following key for our events (they should be ordered by time in ascending order):

eventListName/yyyyMMddHHmmssSSS-000[-111]

where eventListName is the name of the event list and 000 is a three digit instance id to disambiguate between different running instances of application, and -111 is optional to disambiguate events that occured in the same millisecond on one instance.

We additionally insert and artifical row for each day with the id

eventListName/yyyyMMddHHmmssSSS

This allows us to start scanning at the beginning of each day without searching through the event list.

You need to be aware of the fact that if you have a very high load of inserts, then always one hbase region server is busy inserting while the others are idle ... if that's a problem for you, you have to find different keys for your purpose.

You could also use an HBase index table but I have no experience with it and I remember an email on the mailing list that this would double all requests because the API would first lookup the index table and then the original table ??? (please correct me if this is not right ...)

Kind regards, Peter

Thanks Peter.