Using pyparsing to parse a word escape-split over multiple lines

Answer 1

After poking around for a bit more, I came upon this help thread where there was this notable bit

I often see inefficient grammars when someone implements a pyparsing grammar directly from a BNF definition. BNF does not have a concept of "one or more" or "zero or more" or "optional"...

With that, I got the idea to change these two lines

multi_line_word = Forward()
multi_line_word << (word | (split_word + multi_line_word))

To

multi_line_word = ZeroOrMore(split_word) + word

This got it to output what I was looking for: ['super', 'cali', fragi', 'listic'].

Next, I added a parse action that would join these tokens together:

multi_line_word.setParseAction(lambda t: ''.join(t))

This gives a final output of ['supercalifragilistic'].

The take home message I learned is that one doesn't simply walk into Mordor.

Just kidding.

The take home message is that one can't simply implement a one-to-one translation of BNF with pyparsing. Some tricks with using the iterative types should be called into use.

EDIT 2009-11-25: To compensate for the more strenuous test cases, I modified the code to the following:

no_space = NotAny(White(' \t\r'))
# make sure that the EOL immediately follows the escape backslash
continued_ending = Literal('\\') + no_space + lineEnd
word = Word(alphas)
# make sure that the escape backslash immediately follows the word
split_word = word + NotAny(White()) + Suppress(continued_ending)
multi_line_word = OneOrMore(split_word + NotAny(White())) + Optional(word)
multi_line_word.setParseAction(lambda t: ''.join(t))

This has the benefit of making sure that no space comes between any of the elements (with the exception of newlines after the escaping backslashes).

Answer 2

Is this in exercise in pyparsing? If not, then don't bother with pyparsing. Why not just do something like this?

text.replace("\\\n", "")

Answer 3

You are pretty close with your code. Any of these mods would work:

# '|' means MatchFirst, so you had a left-recursive expression
# reversing the order of the alternatives makes this work
multi_line_word << ((split_word + multi_line_word) | word)

# '^' means Or/MatchLongest, but beware using this inside a Forward
multi_line_word << (word ^ (split_word + multi_line_word))

# an unusual use of delimitedList, but it works
multi_line_word = delimitedList(word, continued_ending)

# in place of your parse action, you can wrap in a Combine
multi_line_word = Combine(delimitedList(word, continued_ending))

As you found in your pyparsing googling, BNF->pyparsing translations should be done with a special view to using pyparsing features in place of BNF, um, shortcomings. I was actually in the middle of composing a longer answer, going into more of the BNF translation issues, but you have already found this material (on the wiki, I assume).