Python regex confused by brackets ([])?

Question

Is python confused, or is the programmer?

I've got a lot of lines of this:

some_dict[0x2a] = blah
some_dict[0xab] = blah, blah

What I'd like to do is to convert the hex codes into all uppercase to look like this:

some_dict[0x2A] = blah
some_dict[0xAB] = blah, blah

So I decided to call in the regular expressions. Normally, I'd just do this using my editor's regexps (xemacs), but the need to convert to uppercase pushes one into Lisp. ....ok... how about Python?

So I whip together a short script which doesn't work. I've condensed the code into this example, which doesn't work either. It looks to me like Python's regexps are getting confused by the brackets in the code. Is it me or Python?

import fileinput
import sys
import re


this = "0x2a"
that = "[0x2b]"

for line in [this, that]:
    found = re.match("0x([0-9,a-f]{2})", line)

    if found:
        print("Found: %s" % found.group(0))

(I'm using the () grouping constructs so I don't capitalize the 'x' in '0x'.)

This example only prints the 0x2a value, not the 0x2b. Is this correct behavior?

I can easily work around this by changing the match expression to:

    found = re.match("\[0x([0-9,a-f]{2}\])", line)

but I'm just wondering if someone can give me some insight into what's going on here.

Running Python 2.6.2 on Linux.

Answer 1

re.match matches from the start of the string. Use re.search instead to "match the first occurrence anywhere in the string". The key bit about this in the docs is here.

Answer 2

I don't think you need the comma within the brackets. i.e.:

found = re.match("0x([0-9,a-f]{2})", line)

tells python to look for commas which it might be mistakenly matching. I think you want

found = re.match("0x([0-9a-f]{2})", line)

Answer 3

You want to use re.search. This explains why.

Answer 4

You're using a partial pattern, so you can't use re.match, which expects to match the entire input string. You need to use re.search, which can perform partial matches.

>>> that = "[0x2b]"
>>> m = re.search("0x([0-9,a-f]{2})", that)
>>> m.group()
'0x2b'

Answer 5

You'll want to change

found = re.match("0x([0-9,a-f]{2})", line)

to

found = re.search("0x([0-9,a-f]{2})", line)

re.match will match only from the beginning of the string, which fails in the "[0x2b]" case.

re.search will match anywhere in the string, and thus ignore the leading "[" in the "[0x2b]" case.

See search() vs. match() for details.

Answer 6

If you use re.sub, and pass a callable as the replacement string, it will also do the uppercasing for you:

>>> that = 'some_dict[0x2a] = blah'
>>> m = re.sub("0x([0-9,a-f]{2})", lambda x: "0x"+x.group(1).upper(), that)
>>> m
'some_dict[0x2A] = blah'