What's the cleanest way to extract URLs from a string using Python?

Question

Hi all

Although I know I could use some hugeass regex such as the one posted here I'm wondering if there is some tweaky as hell way to do this either with a standard module or perhaps some third-party add-on?

Simple question, but nothing jumped out on Google (or Stackoverflow).

Look forward to seeing how y'all do this!

Jamie

Answer 1

Use a regular expression.

Reply to comment from the OP: I know this is not helpful. I am telling you the correct way to solve the problem as you stated it is to use a regular expression.

Answer 2

if you know that there is a URL following a space in the string you can do something like this:

s is the string containg the url

>>> t = s[s.find("http://"):]
>>> t = t[:t.find(" ")]

otherwise you need to check if find returns -1 or not.

Answer 3

You can use BeautifulSoup.

def extractlinks(html):
    soup = BeautifulSoup(html)
    anchors = soup.findAll('a')
    links = []
    for a in anchors:
        links.append(a['href'])
    return links

Note that the solution with regexes is faster, although will not be as accurate.

Answer 4

How about your write your own module that implements that regex?

Answer 5

Look at djangos approach here: http://code.djangoproject.com/browser/django/trunk/django/utils/html.py#L77 Regexps are to limited to the job and you have to use heuristics to get results that are mostly right.