How to parse/extract data from a mediawiki marked-up article via python

Question

Source Mediawiki markup

Right now I'm using a variety of regexes to "parse" the data in the mediawiki mark-up into lists/dictionaries, so that elements within the article can be used.

This is hardly the best method, as the number of cases that have to be made are large.

How would one parse an article's mediawiki markup into a variety of python objects so that the data within can be used?

Example being:

• Extract all headlines to a dictionary, hashing it with its section.
• Grab all interwiki links, and stick them into a list (I know
  this can be done from the API but I'd rather only have one API call to
  reduce bandwidth use).
• Extract all image names and hash them with their sections

A variety of regexes can achieve the above, but I'm finding the number I have to make rather large.

Here's the mediawiki unofficial specification (I don't find their official specification as useful).

Answer 1

See PediaPress.

The mwlib Markup Parser generates a semantic parse tree from MediaWiki Markup. This empowers developers to process the vast amount of information available in arbitrary MediaWikis.

The documentation page has a one-liner example:

from mwlib.uparser import simpleparse
simpleparse("=h1=\n*item 1\n*item2\n==h2==\nsome [[Link|caption]] there\n")

If you want to see how it's used in action, see the test cases that come with the code. (mwlib/tests/test_parser.py from git repository):

from mwlib import parser, expander, uparser
from mwlib.expander import DictDB
from mwlib.xfail import xfail
from mwlib.dummydb import DummyDB
from mwlib.refine import util, core

parse = uparser.simpleparse

def test_headings():
    r=parse(u"""
= 1 =
== 2 ==
= 3 =
""")

    sections = [x.children[0].asText().strip() for x in r.children if isinstance(x, parser.Section)]
    assert sections == [u"1", u"3"]

Also see Markup spec and Alternative parsers for more information.