ansaurus

Question

Help with regular expressions (Parsing Wikipedia Markup) in PHP

Answer 1

+1 A:

The current code I am using is the following to clean a Wiki Page, for example this one:

http://en.wikipedia.org/wiki/Tel_Aviv (you can see the markup by clicking "edit this page"

I get this returned:

"and given way to its reputation as a "Mediterranean metropolis that never sleeps". Haaretz Editorial It is the country's financial capital and a major performing arts and business center. Tel Aviv's urban area is the Middle East's second biggest city economy, and is ranked 42nd among global cities by Foreign Policys 2008 Global Cities Index. It is also the most expensive city in the region, and 17th most expensive city in the world. The cost of living in Israel is high, with Tel Aviv being its most expensive city to live in. According to Mercer, a human resources consulting firm based in New York, as of 2008 Tel Aviv is the most expensive city in the Middle East and the 14th most expensive in the world. It falls just behind Singapore and Paris and just ahead of Sydney and Dublin in this respect. By comparison, New York City is 22nd."

Which isn't correct, expected result should be:

Tel Aviv-Yafo (Hebrew: תֵּל־אָבִיב-יָפוֹ; Arabic: تل أبيب‎, Tall ʼAbīb), usually called Tel Aviv, is the second largest city in Israel, with an estimated population of 393,900. The city is situated on the Israeli Mediterranean coastline, with a land area of 51.8 square kilometres (20.0 sq mi). It is the largest and most populous city in the metropolitan area of Gush Dan, home to 3.15 million people as of 2008. The city is governed by the Tel Aviv-Yafo municipality, headed by Ron Huldai.

For this PHP code:

function clean_wiki_text($text)
  {
    // first get rid of UGC HTML tags
    $text = strip_tags($text);

    // keep convert tag
    $text = preg_replace("/\{\{convert\|([^\|]+)\|([^\|]+)\|[^\}]+\}\}/", "$1$2", $text);

    // remove large blocks (treat as tags)
    $text = preg_replace("/(<![^>]+>)/", '', $text);
    $text = preg_replace('/\{\{\s?/', '<', $text);
    $text = str_replace('}}', ' />', $text);

    $text = str_replace('<! />', '', $text);

    // more wiki formatting
    $text = preg_replace("/'{2,6}/", '', $text);
    $text = preg_replace("/[=\s]+External [lL]inks[\s=]+/", '', $text);
    $text = preg_replace("/[=\s]+See [aA]lso[\s=]+/", '', $text);
    $text = preg_replace("/[=\s]+References[\s=]+/", '', $text);
    $text = preg_replace("/[=\s]+Notes[\s=]+/", '', $text);
    $text = preg_replace('/\{\{([^\}]+)\}\}/', '', $text);

    // drop page link text
    $text = preg_replace('/\[\[([^:\|\]]+)\|([^:\]]+)\]\]/', "$2", $text);
    // or keep it with preg_replace('/\[\[([^:\|\]]+)\|([^:\]]+)\]\]/', "$1 ($2)", $text);

    $text = preg_replace('/\(\[[^\]]+\]\)/', '', $text);
    $text = preg_replace('/\[\[([^:\]]+)\]\]/', "$1", $text);
    $text = preg_replace('/\*?\s?\[\[([^\]]+)\]\]/', '', $text);
    $text = preg_replace('/\*\s?\[([^\s]+)\s([^\]]+)\]/', "$2", $text);
    $text = preg_replace('/\n(\*+\s?)/', '', $text);
    $text = preg_replace('/\n{3,}/', "\n\n", $text);
    $text = preg_replace('/<ref[^>]?>[^>]+>/', '', $text);
    $text = preg_replace('/<cite[^>]?>[^>]+>/', '', $text);

    $text = preg_replace('/={2,}/', '', $text);
    $text = preg_replace('/{?class="[^"]+"/', "", $text);
    $text = preg_replace('/!?\s?width="[^"]+"/', "", $text);
    $text = preg_replace('/!?\s?height="[^"]+"/', "", $text);
    $text = preg_replace('/!?\s?style="[^"]+"/', "", $text);
    $text = preg_replace('/!?\s?rowspan="[^"]+"/', "", $text);
    $text = preg_replace('/!?\s?bgcolor="[^"]+"/', "", $text);

    $text = trim($text);

    $text = preg_replace('/\n\n/', "<br />\n<br />\n", $text);
    $text = preg_replace('/\r\n\r\n/', "<br />\r\n<br />\r\n", $text);
/*
    $config = array(
      'show-body-only' => true,
      'clean'          => false, 
      'wrap'           => 0, 
      'show-warnings'  => 0,
      'show-errors'    => 0,
      'enclose-block-text'   => false,
      'vertical-space' => true,
      'output-html'    => true
    );

    // Tidy
    $tidy = new tidy;
    $tidy->parseString($text, $config, 'utf8');
    $tidy->cleanRepair();

    $text = $tidy->value;
*/
    $extras = array(
  //  "/\((.*?)\)/is" => "",
      "/\[(.*?)\]/is" => ""
    );
    $text = preg_replace(array_keys($extras), array_values($extras), $text);

    $text = str_replace(" ,", ',', $text);
    $text = str_replace(", ", ',', $text);
    $text = str_replace(",", ', ', $text);
    $text = str_replace("(, ", '(', $text);
    $text = str_replace(";,", ',', $text);

    // lets keep it plain plain plain
    $text = strip_tags($text);

// $text = preg_replace('/\s\s+/', ' ', $text);

$text = str_replace("|-", '', $text); $text = str_replace("|}", '', $text); $text = str_replace("|", '', $text); $text = str_replace('()', '', $text); $text = str_replace(' ', ' ', $text);

$text = trim($text);

$text_arr = preg_split('/[\r\n]+/', $text, -1, PREG_SPLIT_NO_EMPTY); $result = ""; foreach ($text_arr as $paragraph) { if ( mb_strlen(trim($paragraph)) > 30 ) { $result[] = $paragraph; } } return $result; }

Simon 2010-01-19 17:18:24

Answer 2

A:

It's really hard to make a regex when only one example is provided - from my own experience with cleeaning wikipedia pages I know that other pages will very probably look a little different. Just to match your example is simply:

{{.+?}}\n

This only works if there is a newline after the part to be removed and if you specifiy DOTALL and MULTILINE. Match all pairs of double curly brackets and the stuff inside with:

{{[^}]+}}

You may try to do several runs, each removing another unwanted part - I doubt it's well doable to match all you need inside a single regex.

ferdystschenko 2010-01-19 20:54:13

At first run this, in addition to the below code - for that page does the work, I will test a few other pages - but it looks good.thanks

Simon 2010-01-19 21:34:15

Answer 3

+1 A:

Just guessing here, but wouldn't it be easier and safer to use Wikipedia's markup library (bundled with Mediawiki), turn that into HTML then parse it using whatever XML library you happen to be comfortable with ?

API documentation can be found at http://svn.wikimedia.org/doc/ (in the Parser module) and it doesn't look very complicated. Basically, all you'd have to do is something like the following:

<?php

require_once '/path/to/mediawiki/Parser.php';
// also include whatver classes Parser depends on or use Mediawiki's autoload
// mechanism if it has any

// retrieve the content of your page in $content

$parser = new Parser();
$html   = $parser->parse($content);

$simplexml = simplexml_load_string($html);

Now you have a very handy SimpleXML object to play with. Of course, this only works if Mediawiki's parser produces valid XML (which I bet it does).

Also, should Mediawiki include some kind of autoload mechanism, it would be easy to find it by looking for __autoload or spl_autoload_register in Mediawiki's codebase.

Hope it helps!

Geoffrey Bachelet 2010-01-27 00:26:27

ansaurus

tags:

views:

answers:

Help with regular expressions (Parsing Wikipedia Markup) in PHP

related questions