How to build large/busy RSS feed

Question

I've been playing with RSS feeds this week, and for my next trick I want to build one for our internal application log. We have a centralized database table that our myriad batch and intranet apps use for posting log messages. I want to create an RSS feed off of this table, but I'm not sure how to handle the volume- there could be hundreds of entries per day even on a normal day. An exceptional make-you-want-to-quit kind of day might see a few thousand. Any thoughts?

Answer 1

Without knowing your application, I can't offer specific advice.

That said, it's common in these sorts of systems to have a level of severity. You could have a query string parameter that you tack on to the end of the URL that specifies the severity. If set to "DEBUG" you would see every event, no matter how trivial. If you set it to "FATAL" you'd only see the events that that were "System Failure" in magnitude.

If there are still too many events, you may want to sub-divide your events in to some sort of category system. Again, I would have this as a query string parameter.

You can then have multiple RSS feeds for the various categories and severities. This should allow you to tune the level of alerts you get an acceptable level.

Answer 2

Okay, I decided how I'm gonna handle this. I'm using the timestamp field for each column and grouping by day. It takes a little bit of SQL-fu to make it happen since of course there's a full timestamp there and I need to be semi-intelligent about how I pick the log message to show from within the group, but it's not too bad. Further, I'm building it to let you select which application to monitor, and then showing every message (max 50) from a specific day.

That gets me down to something reasonable.

I'm still hoping for a good answer to the more generic question: "How do you syndicate many important messages, where missing a message could be a problem?"

Answer 3

If you are building a system with notifications that must not be missed, then a pub-sub mechanism (using XMPP, one of the other protocols supported by ApacheMQ, or something similar) will be more suitable that a syndication mechanism. You need some measure of coupling between the system that is generating the notifications and ones that are consuming them, to ensure that consumers don't miss notifications.

(You can do this using RSS or Atom as a transport format, but it's probably not a common use case; you'd need to vary the notifications shown based on the consumer and which notifications it has previously seen.)

Answer 4

In this case, it's more of a manager's dashboard: how much work was put into support today, is there anything pressing in the log right now, and for when we first arrive in the morning as a measure of what went wrong with batch jobs overnight.

Answer 5

I would make the feed a static file (you can easily serve thousands of these), regenerated periodically. Then you have a much broader choice, because it doesn't have to run below second, it can run even minutes. And users still get perfect download speed and reasonable update speed.

Answer 6

I'd split up the feeds as much as possible and let users recombine them as desired. If I were doing it I'd probably think about using Django and the syndication framework.

Django's models could probably handle representing the data structure of the tables you care about.

You could have a URL that catches everything, like: r'/rss/(?(\w*?)/)+' (I think that might work, but I can't test it now so it might not be perfect).

That way you could use URLs like (edited to cancel the auto-linking of example URLs):

• http:// feedserver/rss/batch-file-output/
• http:// feedserver/rss/support-tickets/
• http:// feedserver/rss/batch-file-output/support-tickets/ (both of the first two combined into one)

Then in the view:

def get_batch_file_messages():
    # Grab all the recent batch files messages here.
    # Maybe cache the result and only regenerate every so often.

# Other feed functions here.

feed_mapping = { 'batch-file-output': get_batch_file_messages, }

def rss(request, *args):
    items_to_display = []
    for feed in args:
        items_to_display += feed_mapping[feed]()
    # Processing/returning the feed.

Having individual, chainable feeds means that users can subscribe to one feed at a time, or merge the ones they care about into one larger feed. Whatever's easier for them to read, they can do.