Scraping html with Python or...

Question

One of the arguments I make to my (Microbiology and Genetics) students is that "data" is/are messy, and Python can help with that (of course other languages can too). So here is a practical kind of web-based data-gathering exercise.

I notice that there a few people who answer Python-related questions among the users with the highest rep. Among the questions which naturally arise:

I wish to recover the current rep and rate of rep increase for (top rated) Pythonistas on Stack Overflow, so as to predict whether or when Alex Martelli will overtake Steven Lott or Greg Hewgill? what about Konrad Rudolph? Is it trivial because the increases for these guys are all pegged to the limit?

More generally, in the absence of an API for querying (which I don't think there is) is there any alternative to looking through the URLs of pages for patterns, loading those pages with Python and then scraping the html? I realize there is probably not a general approach, but I'm interested in how people would approach this problem.

Edit: @fitzgeraldsteele: Generally. SO is really just a (contrived) example.

Answer 1

There's a perfectly usable monthly "data dump" of Stack Overflow under Creative Commons license, see for example here (just the first one "under my thumb" of the many links about this -- at least one per month). For such analysis as my average weekly rep relative to some other poster's, such monthly dollops of data are much more usable than screen-scrapes.

If you do want to screen-scrape some (other;-) site, and that doesn't violate their policies or their robots.txt files, Python is one of several excellent choices -- start with scrapy, and you won't have all that much extra work to do, for example.