How to find links and modify an Html using BeautifulSoup in Python

Question

Starting from an Html input like this:

<p>
<a href="http://www.foo.com"&gt;this if foo</a>
<a href="http://www.bar.com"&gt;this if bar</a>
</p>

using BeautifulSoup, i would like to change this Html in:

<p>
<a href="http://www.foo.com"&gt;this if foo[1]</a>
<a href="http://www.bar.com"&gt;this if bar[2]</a>
</p>

saving parsed links in a dictionary with a result like this:

links_dict = {"1":"http://www.foo.com","2":"http://www.bar.com"}

Is it possible to do this using BeautifulSoup? Any valid alternative?

Answer 1

This should be easy in Beautiful Soup.

Something like:

from BeautifulSoup import BeautifulSoup
from BeautifulSoup import Tag

count = 1
links_dict = {}
soup = BeautifulSoup(text)
for link_tag in soup.findAll('a'):
  if link_tag['href'] and len(link_tag['href']) > 0:
    links_dict[count]  = link_tag['href']  
    newTag = Tag(soup, "a", link_tag.attrs)
    newTag.insert(0, ''.join([''.join(link_tag.contents), "[%s]" % str(count)]))
    link_tag.replaceWith(newTag)
    count += 1

Result of executing this on your text:

>>> soup
<p>
  <a href="http://www.foo.com"&gt;this if foo[1]</a>
  <a href="http://www.bar.com"&gt;this if bar[2]</a>
</p>

>>> links_dict
{1: u'http://www.foo.com', 2: u'http://www.bar.com'}

The only problem I can foresee with this solution is if your link text contains subtags; then you couldn't do ''.join(link_tag.contents); instead you would need to navigate to the rightmost text element.