Optimising RSS parsing on App Engine to avoid high CPU warnings

Question

I'm pulling some RSS feeds into a datastore in App Engine to serve up to an iPhone app. I use cron to schedule updating the RSS every x minutes. Each task only parses one RSS feed (which has 15-20 items). I frequently get warnings about high CPU usage in the App Engine dashboard, so I'm looking for ways to optimise my code.

Currently, I use minidom (since it's already there on App Engine), but I suspect it's not very efficient!

Here's the code:

 dom = minidom.parseString(urlfetch.fetch(url).content)
    if dom:
        items = []
        for node in dom.getElementsByTagName('item'):
            item = RssItem(
                key_name = self.getText(node.getElementsByTagName('guid')[0].childNodes),
                title = self.getText(node.getElementsByTagName('title')[0].childNodes),
                description = self.getText(node.getElementsByTagName('description')[0].childNodes),
                modified = datetime.now(),
                link = self.getText(node.getElementsByTagName('link')[0].childNodes),
                categories = [self.getText(category.childNodes) for category in node.getElementsByTagName('category')]
            );
            items.append(item);
        db.put(items);

def getText(self, nodelist):
    rc = ''
    for node in nodelist:
        if node.nodeType == node.TEXT_NODE:
            rc = rc + node.data
    return rc

There isn't much going on, but the scripts often take 2-6 seconds CPU time, which seems a bit excessive for looping through 20ish items and reading a few attributes.

What can I do to make this faster? Is there anything particularly bad in the above code, or should I change to another way of parsing? Are there are any libraries (that work on App Engine) that would be better, or would I be better parsing the RSS myself?

Answer 1

I'd try ElementTree or the Universal Feed Parser and see if they're any better. ElementTree is in the stdlib as of Python 2.5, so it's available on App Engine.

Answer 2

You probably should run a profiler to pinpoint where the code is spinning its wheels. It could be waiting on the connections as some RSS feeds are REAL slow.

Also, some RDF/RSS/ATOM libraries build in a governor to keep from beating the cr*p out of the host when retrieving multiple feeds from the same site. I've written several aggregators and being considerate to the server is important.

Universal Feed Parser is full-featured, at least from what I've seen by looking through the docs. I didn't use it because I wrote my aggregators in Ruby and had different needs but I was aware of it and would consider it for a Python-based solution.

Answer 3

Outsource feed parsing via for example superfeedr

You could also look into superfeedr.com. They have a reasonable free quota/paying plans. They will do the polling(within 15 minutes you get updates) for you/etc. If the feeds also support pubsubhubbub, then you will receive the feeds in realtime! This video will explain to you what pubsubhubbub is if you don't know yet.

Improved feed parser written by Brett Slatkin

I would also advice you to watch this awesome video from Brett Slatkin explaining pubsubhubbub. I also remember that somewhere in the presentation he says that he does not use Universal Feedparser because it's just does to much work for his problem. He wrote his own SAX(14:10 in video presentation he talks about it a little bit) parser which is lightning fast. I guess you should check out the pubsubhubbub code to find out how he accomplished this.

Answer 4

If you have a low amount of traffic coming to your site you might be experiencing spin up times for your app. If an app is idle for a as little as a few minutes app engine will spin down your app to save resources. When the next request comes in the app has to be spun up before it can handle the request and this all gets added to your cpu quota. If you search the appengine newsgroup you see that it is full of complaints about this.

I use superfeedr for my site www.newsfacet.com and I notice that when superfeedr notifies me most of the time I can handle a few rss articles in a few hundred milliseconds. If its been a while since the last input this time can jump to 10 or 11 seconds as it incurs the spin up cost.

Answer 5

In regards to using PubSubHubbub to let someone else do the work for you, you may find my blog post on using hubbub on App Engine to be useful.