Pythonic way to find a regular expression match

Question

Is there a more succinct/correct/pythonic way to do the following:

url = "http://0.0.0.0:3000/authenticate/login"
re_token = re.compile("<[^>]*authenticity_token[^>]*value=\"([^\"]*)")
for line in urllib2.urlopen(url):
    if re_token.match(line):
        token = re_token.findall(line)[0]
        break

I want to get the value of the input tag named "authenticity_token" from an HTML page:

<input name="authenticity_token" type="hidden" value="WTumSWohmrxcoiDtgpPRcxUMh/D9m7O7T6HOhWH+Yw4=" />

Answer 1

Could you use Beautiful Soup for this? The code would essentially look something like so:

from BeautifulSoup import BeautifulSoup
url = "hhttp://0.0.0.0:3000/authenticate/login"
page = urlli2b.urlopen(page)
soup = BeautifulSoup(page)
token = soup.find("input", { 'name': 'authenticity_token'})

Something like that should work. I didn't test this but you can read the documentation to get it exact.

Answer 2

You don't need the findall call. Instead use:

m = re_token.match(line)
if m:
    token = m.group(1)
    ....

I second the recommendation of BeautifulSoup over regular expressions though.

Answer 3

there's nothing "pythonic" with using regex. If you don't want to use BeautifulSoup(which you should ideally), just use Python's excellent string manipulation capabilities

for line in open("file"):
    line=line.strip()
    if "<input name" in line and "value=" in line:
        item=line.split()
        for i in item:
            if "value" in i:
                print i

output

$ more file
<input name="authenticity_token" type="hidden" value="WTumSWohmrxcoiDtgpPRcxUMh/D9m7O7T6HOhWH+Yw4=" />
$ python script.py
value="WTumSWohmrxcoiDtgpPRcxUMh/D9m7O7T6HOhWH+Yw4="

Answer 4

As to why you shouldn't use regular expressions to search HTML, there are two main reasons.

The first is that HTML is defined recursively, and regular expressions, which compile into stackless state machines, don't do recursion. You can't write a regular expression that can tell, when it encounters an end tag, what start tag it encountered on its way to that tag it belongs to; there's nowhere to save that information.

The second is that parsing HTML (which BeautifulSoup does) normalizes all kinds of things that are allowable in HTML and that you're probably not going to ever consider in your regular expressions. To pick a trivial example, what you're trying to parse:

<input name="authenticity_token" type="hidden" value="xxx"/>

could just as easily be:

<input name='authenticity_token' type="hidden" value="xxx"/>

or

<input type = "hidden" value = "xxx" name = 'authenticity_token' />

or any one of a hundred other permutations that I'm not thinking about right now.