Scraping and Parsing a Wikipedia Page

Question

Hey guys. I'm wondering if there are any existing libraries in or accessible from Objective-C that would allow me to scrape pages formatted like this one. Specifically, all of the dates and all of the text next to each date. If not, what would be the best way to go about doing this? Regular expressions? I heard that NSString might already have built-in methods for this. Is this true?

I was looking around to see if there were any alternative to scraping, such as an XML file or API. I did find an API but the only clients I see available are in other languages and they seem to just be able to post content to pages, not retrieve it.

EDIT: So I found more information regarding the API at these links:

• MediaWiki API
• API:Query

And I was able to come up with this request which returns some HTML encoded text (Well the format is XML, but it includes the page's text such as »a href= etc. I'll keep looking through the docs to see if I can make this come out a bit better, if not though, are there any recommendations on parsing this?

EDIT 2: Alright so thanks to this doc page, the simplest and cleanest way I've been able to retrieve the data is using this constructed link which returns the raw data (In wiki markup) of the relevant section. However, I guess I would then need to parse that, though if that really is the case, it should be a lot easier than the entire article.

Does anyone have any recommendations on parsing wiki markup such as the following in Objective-C?

==Events==
* [[710]] – [[Saracen]] invasion of [[Sardinia]].
*[[1275]] – Traditional founding of the city of [[Amsterdam]].
*[[1682]] – [[Philadelphia]], [[Pennsylvania]] is founded.

What I want to end up having is, I guess an NSDictionary or similar collection that will store the date with the accompanying snippet of information. Thanks!

Answer 1

that's most definitely not the way to do it, in any language.

if any site online will expose their data in a nice way, it'll be wikipedia.

look into getting an article as XML, as RDF, or maybe even as JSON.

Answer 2

I'm going to go with suggesting regex for targeted data extraction in a mixed HTML data stream.

There are already RegEx libraries on the phone, they are sort of hidden though - you can expose them with a few simple calls using RegexKitLite (make sure to scroll down and get the light version). It ends up being a class with a few extensions on NSString that lets you do regexs, then you would define a regex with two captured matches - one for the number, and one for the content, along with a number of non-captured matches for the enclosing and intermediate tags. Even though it's a "lite" version of standard RegEX it sill supports just about any ability you would need.

The API approach is promising but once you get the raw markup you're probably going to have to take a similar regex approach to parsing data out of that. It still might make sense if it reduces regex complexity and data transfer time though, no reason you can't combine both approaches.

Answer 3

Given that pages on Wikipedia are stored as plaintext, and input by users as plaintext, you're not going to get a structured data set from it.

Answer 4

Add a &format=fmt to the end of your query, as described at API:Data_formats. Your query becomes: JSON query, for example. You can specify XML, JSON, or many other formats.

You can easily parse the overall sections, and then just display the HTML formatted output into a webview.

Answer 5

I have scraped a lot of data from WP in various ways. the format depends on a lot of things including what type of subdomain the information is in and when it was entered. The main text is free format and there is no simple way to scrape it. The infoboxes are in a special WP format which has changed over the years. It wasn't designed to be scraped.

There is a database backing WP which is somewhat more structured.

By far your best strategy is to contact the Wikipedians in the domain you wish to scrape - they will know about the database format and may well be able to help - they will certainly want to help as they will want to see WP in semantic form (such as DBPedia - http://dbpedia.org/About).

Answer 6

Does Python count? ;) It is accessible from Objective-C. And there are great modules for scraping purposes: Beautiful Soap and/or mechanize, you can also consider lxml.

Answer 7

I've got an iPhone app which does a screen scrape using the following:

• YQL (http://developer.yahoo.com/yql)
• Yahoo's Objective-C Libraries (http://github.com/yahoo/yos-social-objc)

Using YQL you can get whatever information you need from the web by using XPATH queries against the DOM.

Personally I think its much better than using Regex. Then again I only know very simple regular expressions.