Regex for parsing Wikicode in C#

Question

Hi folks,

I try to parse articles from wikipedia. I use the *page-articles.xml file, where they backup all their articles in a wikicode-format. To strip the format and get the raw text, I try to use Regular Expressions, but I am not very used to it. I use C# as programming language. I tried a bit around with Expresso, a designer for Regular Expressions, but I am at the end of my wits. Here is what I want to achieve:

The text can contain the following structures: [[TextN]] or [[Text1|TextN]] or [[Text1|Text2|...|TextN]]

the [[ .... ]] pattern can appear within the Texti aswell. I want to replace these structure with TextN

For identifing the structures withhin the text I tried the following RegEx:

\[\[ ( .* \|?)* \]\]

Expresso seems to run and endless loop with this one. After 5 minutes for a relative small text, I canceled the Test Run.

Then I tried something more simple, I want to capture anything between the brackets:

\[\[ .* \]\]

but when I have a line like:

[[Word1]] text inbetween [[Word2]]

the expression returns the whole line, not

[[Word1]]

[[Word2]]

Any tips from Regex-Experts here to solve the problem?

Thanks in advance, Frank

Answer 1

This is because regular expressions tries to find always the longest matches possible. You should change .*

Try using

\[\[([A-Za-z][A-Za-z\d+]*)(\|\1)*\]\]

This will match only letters, | sign and numbers in double brackets + it checks if value starts with the letter.

Answer 2

\[\[(.*?\]\] would do it.

The key is the .*? which means get any characters but as few a possible.

EDIT

For nested tags one approach would be:

\[\[(?<text>(?>\[\[(?<Level>)|\]\](?<-Level>)|(?! \[\[ | \]\] ).)+(?(Level)(?!)))\]\]

This ensures that the [[ and ]] match across the text as well.

Answer 3

If Expresso isn't working out for you, you may want to try RegexBuddy.

While not free, it does provide an excellent real time testing environment where you can see how your regex is going to match a section of sample text.

Answer 4

I wouldn't use regular expressions (since they don't handle recursion/nesting well).

Instead I would parse the text by hand*, which isn't particularly difficult in this case.

You could represent the text as a stream of elements whereas each element is either

• a plain text chunk, or
• a tag

A tag might contain multiple element streams, separated by |.

elementStream ::= element*
element ::= chunk | tag
chunk ::= TEXT
tag ::= "[[" elementStream otherStreams "]]"
otherStreams ::= "|" elementStream otherStreams

Your parser could represent each of those definitions with a method. So you'd have an elementStream method that would call element as long as there is text available and the next two characters are not "]]" or "|" (if you are inside a tag). Each call to element would return the element parsed, either a chunk or a tag. etc.

This would essentially be a recursive descent parser. Wikipedia: http://en.wikipedia.org/wiki/Recursive_descent_parser (the article is rather long/complicated, unfortunately)

Answer 5

If GPL2 is not an issue for you, maybe you could check out the source code of Screwturn Wiki and see how an expert does it. It's in C#, BTW