What is the best way to handle a bad link given to BeautifulSoup?

Question

I'm working on something that pulls in urls from delicious and then uses those urls to discover associated feeds.

However, some of the bookmarks in delicious are not html links and cause BS to barf. Basically, I want to throw away a link if BS fetches it and it does not look like html.

Right now, this is what I'm getting.

trillian:Documents jauderho$ ./d2o.py "green data center" 
processing http://www.greenm3.com/
processing http://www.eweek.com/c/a/Green-IT/How-to-Create-an-EnergyEfficient-Green-Data-Center/?kc=rss
Traceback (most recent call last):
  File "./d2o.py", line 53, in <module>
    get_feed_links(d_links)
  File "./d2o.py", line 43, in get_feed_links
    soup = BeautifulSoup(html)
  File "/Library/Python/2.5/site-packages/BeautifulSoup.py", line 1499, in __init__
    BeautifulStoneSoup.__init__(self, *args, **kwargs)
  File "/Library/Python/2.5/site-packages/BeautifulSoup.py", line 1230, in __init__
    self._feed(isHTML=isHTML)
  File "/Library/Python/2.5/site-packages/BeautifulSoup.py", line 1263, in _feed
    self.builder.feed(markup)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/HTMLParser.py", line 108, in feed
    self.goahead(0)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/HTMLParser.py", line 150, in goahead
    k = self.parse_endtag(i)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/HTMLParser.py", line 314, in parse_endtag
    self.error("bad end tag: %r" % (rawdata[i:j],))
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/HTMLParser.py", line 115, in error
    raise HTMLParseError(message, self.getpos())
HTMLParser.HTMLParseError: bad end tag: u'</b  />', at line 739, column 1

Update:

Jehiah's answer does the trick. For reference, here's some code to get the content type:

def check_for_html(link):
    out = urllib.urlopen(link)
    return out.info().getheader('Content-Type')

Answer 1

I simply wrap my BeautifulSoup processing and look for the HTMLParser.HTMLParseError exception

import HTMLParser,BeautifulSoup
try:
    soup = BeautifulSoup.BeautifulSoup(raw_html)
    for a in soup.findAll('a'):
        href = a.['href']
        ....
except HTMLParser.HTMLParseError:
    print "failed to parse",url

but further than that, you can check the content type of the responses when you crawl a page and make sure that it's something like text/html or application/xml+xhtml or something like that before you even try to parse it. That should head off most errors.