python: replacing regex with BNF or pyparsing

Question

I am parsing a relatively simple text, where each line describes a game unit. I have little knowledge of parsing techniques, so I used the following ad hoc solution:

class Unit:
    # rules is an ordered dictionary of tagged regex that is intended to be applied in the given order
    # the group named V would correspond to the value (if any) for that particular tag
    rules = (
        ('Level', r'Lv. (?P<V>\d+)'),
        ('DPS', r'DPS: (?P<V>\d+)'),
        ('Type', r'(?P<V>Tank|Infantry|Artillery'),
        #the XXX will be expanded into a list of valid traits
        #note: (XXX| )* wouldn't work; it will match the first space it finds,
        #and stop at that if it's in front of something other than a trait
        ('Traits', r'(?P<V>(XXX)(XXX| )*)'),
        # flavor text, if any, ends with a dot
        ('FlavorText', r'(?P<V>.*\."?$)'),
        )
    rules = collections.OrderedDict(rules)
    traits = '|'.join('All-Terrain', 'Armored', 'Anti-Aircraft', 'Motorized')
    rules['Traits'] = re.sub('XXX', effects, rules['Traits'])

    for x in rules:
        rules[x] = re.sub('<V>', '<'+x+'>', rules[x])
        rules[x] = re.compile(rules[x])

    def __init__(self, data)
        # data looks like this:
        # Lv. 5 Tank DPS: 55 Motorized Armored
        for field, regex in Item.rules.items():
            data = regex.sub(self.parse, data, 1)
        if data:
            raise ParserError('Could not parse part of the input: ' + data)

    def parse(self, m):
        if len(m.groupdict()) != 1:
            Exception('Expected a single named group')
        field, value = m.groupdict().popitem()
        setattr(self, field, value)
        return ''

It works fine, but I feel I reached the limit of regex power. Specifically, in the case of Traits, the value ends up being a string that I need to split and convert into a list at a later point: e.g., obj.Traits would be set to 'Motorized Armored' in this code, but in a later function changed to ('Motorized', 'Armored').

I'm thinking of converting this code to use either EBNF or pyparsing grammar or something like that. My goals are:

• make this code neater and less error-prone
• avoid the ugly treatment of the case with a list of values (where I need do replacement inside the regex first, and later post-process the result to convert a string into a list)

What would be your suggestions about what to use, and how to rewrite the code?

P.S. I skipped some parts of the code to avoid clutter; if I introduced any errors in the process, sorry - the original code does work :)

Answer 1

I started to write up a coaching guide for pyparsing, but looking at your rules, they translate pretty easily into pyparsing elements themselves, without dealing with EBNF, so I just cooked up a quick sample:

from pyparsing import Word, nums, oneOf, Group, OneOrMore, Regex, Optional

integer = Word(nums)
level = "Lv." + integer("Level")
dps = "DPS:" + integer("DPS")
type_ = oneOf("Tank Infantry Artillery")("Type")
traits = Group(OneOrMore(oneOf("All-Terrain Armored Anti-Aircraft Motorized")))("Traits")
flavortext = Regex(r".*\.$")("FlavorText")

rule = (Optional(level) & Optional(dps) & Optional(type_) & 
        Optional(traits) & Optional(flavortext))

I included the Regex example so you could see how a regular expression could be dropped in to an existing pyparsing grammar. The composition of rule using '&' operators means that the individual items could be found in any order (so the grammar takes care of the iterating over all the rules, instead of you doing it in your own code). Pyparsing uses operator overloading to build up complex parsers from simple ones: '+' for sequence, '|' and '^' for alternatives (first-match or longest-match), and so on.

Here is how the parsed results would look - note that I added results names, just as you used named groups in your regexen:

data = "Lv. 5 Tank DPS: 55 Motorized Armored"

parsed_data = rule.parseString(data)
print parsed_data.dump()
print parsed_data.DPS
print parsed_data.Type
print ' '.join(parsed_data.Traits)

prints:

['Lv.', '5', 'Tank', 'DPS:', '55', ['Motorized', 'Armored']]
- DPS: 55
- Level: 5
- Traits: ['Motorized', 'Armored']
- Type: Tank
55
Tank
Motorized Armored

Please stop by the wiki and see the other examples. You can easy_install to install pyparsing, but if you download the source distribution from SourceForge, there is a lot of additional documentation.