views:

705

answers:

5

I currently have some Ruby code used to scrape some websites. I was using Ruby because at the time I was using Ruby on Rails for a site, and it just made sense.

Now I'm trying to port this over to Google App Engine, and keep getting stuck.

I've ported Python Mechanize to work with Google App Engine, but it doesn't support DOM inspection with XPATH.

I've tried the built-in ElementTree, but it choked on the first HTML blob I gave it when it ran into '&mdash'.

Do I keep trying to hack ElementTree in there, or do I try to use something else?

thanks, Mark

+11  A: 

Beautiful Soup.

S.Lott
For some reason I was thinking that was pure python, but it looks like it is. I'll check it out.
MStodd
Second that. Beautiful Soup is incredible.
David Wolever
+1 for Beautiful Soup. Scraping is its whole purpose.
steveha
Right answer, but there's something fundamentally broken about getting 60+ points for being the first person to write two words. ;)
Nick Johnson
@Nick Johnson: Since it's a duplicate question, it's doubly wrong to get upvoted for answering it yet again.
S.Lott
It's not really a dup. I need a pure python solution that works with XPath. I don't know that any suggestions so far meet those requirements.
MStodd
+4  A: 

lxml -- 100x better than elementtree

Billy Joe
lxml is a wrapper for a C library, so it cannot run on appengine.
Roberto Bonvallet
It's also going to barf just as hard on badly formed HTML.
jcdyer
jcd - not true. lxml includes several options for parsing HTML, including using BeautifulSoup as a parser backend - http://codespeak.net/lxml/elementsoup.html
Matt Good
+2  A: 

There's also scrapy, might be more up your alley.

Autoplectic
+1 to scrapy. Works really well.
nosklo
A: 

There are a number of examples of web page scrapers written using pyparsing, such as this one (extracts all URL links from yahoo.com) and this one (for extracting the NIST NTP server addresses). Be sure to use the pyparsing helper method makeHTMLTags, instead of just hand coding "<" + Literal(tagname) + ">" - makeHTMLTags creates a very robust parser, with accommodation for extra spaces, upper/lower case inconsistencies, unexpected attributes, attribute values with various quoting styles, and so on. Pyparsing will also give you more control over special syntax issues, such as custom entities. Also it is pure Python, liberally licensed, and small footprint (a single source module), so it is easy to drop into your GAE app right in with your other application code.

Paul McGuire
A: 

BeautifulSoup is good, but its API is awkward. Try ElementSoup, which provides an ElementTree interface to BeautifulSoup.

Plumo