Can You Use a Single Regular Expression to Parse Function Parameters?

Question

Problem

There is a program file that contains the following code snippet at some point in the file.

...

food($apples$ , $oranges$ , $pears$ , $tomato$){
  ...
}

...

This function may contain any number of parameters but they must be strings separated by commas. All the parameter strings are lowercase words.

I want to be able to parse out each of the parameters using a regular expression. For example the resulting list in python would be as follows:

["apples", "oranges", "pears", "tomato"]

Attempted Solution

Using the python RE module, I was able to achieve this by breaking the problem into two parts.

1. Find the function in the code and extract the list of parameters.

   plist = re.search(r'food\((.*)\)', programString).group(1)
   

2. Split the list using another regular expression.

   params = re.findall(r'[a-z]+', plist)
   

Question

Is there anyway I could achieve this with one regular expression instead of two?

Edit

Thanks to Tim Pietzcker's answer I was able to find some related questions:

1. http://stackoverflow.com/questions/464736/python-regular-expressions-how-to-capture-multiple-groups-from-a-wildcard-expre
2. http://stackoverflow.com/questions/2652554/which-regex-flavors-support-captures-as-opposed-to-capturing-groups

Answer 1

params = re.findall(r'\$([a-z]+)\$', programString)

Answer 2

Something like this regex should work

food\((\$(?<parm>\w+)\$\s*,?\s*)+\).*

it puts all the matching parameter names in the 'parm' group

Answer 3

Pyparsing is handy for this kind of thing, when you don't know when you'll run into extra whitespace, comments, whatever. Like named groups in RE, this example defines the results name 'parameters' which is used to retrieve the desired data:

>>> code = """\
... ...
...
... food($apples$ , $oranges$ , $pears$ , $tomato$){
...   ...
... }
... ...
... food($peanuts$, $popcorn$ ,$candybars$ ,$icecream$){
...   ...
... }
... """
>>> from pyparsing import *
>>> LPAR,RPAR,LBRACE,RBRACE,DOLLAR = map(Suppress,"(){}$")
>>> param = DOLLAR + Word(alphas) + DOLLAR
>>> funcCall = "food" + LPAR + delimitedList(param)("parameters") + RPAR + LBRACE
>>> for fn in funcCall.searchString(code):
...   print fn.parameters
...
['apples', 'oranges', 'pears', 'tomato']
['peanuts', 'popcorn', 'candybars', 'icecream']

If I change the second function to:

... food($peanuts$, $popcorn$ ,/*$candybars$ ,*/$icecream$){

And then add this line:

>>> funcCall.ignore(cStyleComment)

Then I get:

>>> for fn in funcCall.searchString(code):
...   print fn.parameters
...
['apples', 'oranges', 'pears', 'tomato']
['peanuts', 'popcorn', 'icecream']

Answer 4

To answer your question "Can it be done in a single regex?": Yes, but not in Python.

If you want to match and capture (individually) an unknown number of matches as in your example, using only a single regular expression, then you need a regex engine that supports captures (as opposed to capturing groups). Only .NET and Perl 6 do this currently.

So in Python, you either need to do it in two steps (find the entire food(...) function call, and then findall individual matches with a second regex as suggested by Dingo).

Or use a parser like Paul McGuire's pyparsing.

Answer 5

Why regex?

for line in open("file"):
    line=line.rstrip()
    if line.lstrip().startswith("food") :
        for item in line.split(")"):
            if "food" in item:
                print item.split("(")[-1].split(",")

output

$ ./python.py
['$apples$ ', ' $oranges$ ', ' $pears$ ', ' $tomato$']