Scraping digg rss feed with python

Question

is there a way to get the link from digg through its rss feed? or do i have to get the website and manually scrape it with a regex?

i want to get the real link digg points to, not to the comments feed, from rss.

example - http://feeds.digg.com/~r/digg/popular/~3/Hx0VATaafSw/Apple_Scaling_Final_Cut_Studio_Apps_to_Fit_Prosumers_2

goes to

http://www.appleinsider.com/articles/10/05/18/apple_scaling_final_cut_studio_apps_to_fit_prosumers.html

Answer 1

Take a look at the YQL @ Yahoo...

Here is a query that returns XML from digg

http://developer.yahoo.com/yql/console/?q=select%20title%2Clink%20from%20rss%20where%20url%3D%22http%3A%2F%2Ffeeds.digg.com%2Fdigg%2Fpopular.rss%22

You can parse JSON or XML.

Good Luck!

Answer 2

Take a look at the feedparser module.

>>> import feedparser
>>> d = feedparser.parse('http://feeds.digg.com/digg/popular.rss')
>>> for entry in d.entries:
...     print entry.link
...
http://feeds.digg.com/~r/digg/popular/~3/Hx0VATaafSw/Apple_Scaling_Final_Cut_Studio_Apps_to_Fit_Prosumers_2
http://feeds.digg.com/~r/digg/popular/~3/mXb8b0QH3Rc/Skateboarder_Lives_Any_Man_s_Worst_Nightmare_video
http://feeds.digg.com/~r/digg/popular/~3/61N9gFUth1k/CBS_A_bloodbath_of_cancellations
http://feeds.digg.com/~r/digg/popular/~3/vZ3_6F1RAcI/Red_Dead_Redemption_Free_Roam_Done_Right
(snip)

Answer 3

It looks like you will need to use the digg api to get the actual links to the stories, and not just the link to the digg comments. The API can give you data in XML or JSON, both of which are easily handled in python -- lxml and simplejson both work well.

The other option, if you are really keen to using the RSS feeds, is to parse the digg links and then scrape the links off of that page -- but that is going to be less efficient and more prone to breaking.

I've run into this issue on similar social news and blog sites -- basically they want you to land on their page before you go off to read the actual story. Understandable, but kind of annoying from a scripting point-of-view.

Answer 4

You can use the story.getInfo method of the Digg API. One of its possible arguments is clean_title which you can parse from the link in the RSS feed. Here's a sample implementation:

import feedparser
import urllib2
from xml.etree import ElementTree

rss_link = 'http://feeds.digg.com/digg/popular.rss'
api_link = 'http://services.digg.com/1.0/endpoint?method=story.getInfo&clean_title=%s'

data = feedparser.parse(rss_link)

for i, e in enumerate(data.entries, 1):
  print '%d. Digg link: %s' % (i, e.link)
  title = e.link[e.link.rfind('/') + 1 :]
  xml = urllib2.urlopen(api_link % title).read()
  tree = ElementTree.fromstring(xml)
  print '%d. Real link: %s' % (i, tree.find('story').get('link'))

... which outputs:

1. Digg link: http://feeds.digg.com/~r/digg/popular/~3/V58R-d7nd2M/Pakistan_court_bans_Facebook_site
1. Real link: http://news.bbc.co.uk/2/hi/south_asia/8691406.stm
2. Digg link: http://feeds.digg.com/~r/digg/popular/~3/LoF6h1fTtk/Britons_spend_more_webtime_reading_news_than_looking_at_porn
2. Real link: http://www.telegraph.co.uk/technology/news/7740500/Britons-spend-more-web-time-reading-news-than-looking-at-pornography.html
3. Digg link: http://feeds.digg.com/~r/digg/popular/~3/XQUD2tR-qGQ/Sludgy_oil_begins_washing_into_Lousiana_s_coastal_marshes
3. Real link: http://www.washingtonpost.com/wp-dyn/content/article/2010/05/18/AR2010051801676.html?hpid=topnews
4. Digg link: http://feeds.digg.com/~r/digg/popular/~3/4HBB7lvCpoM/Professor_examines_the_complex_evolution_of_human_morality
4. Real link: http://www.physorg.com/news193472479.html
5. Digg link: http://feeds.digg.com/~r/digg/popular/~3/9__2-MVmSp4/How_Are_America_s_Top_Companies_Taxed_Infographic
5. Real link: http://www.mint.com/blog/trends/how-are-americas-top-companies-taxed/
...