Best methods to parse HTML

Question

Im working on a system that requires the parsing of HTML documents under PHP.

my question is simply this:

What's the best method of parsing content for relative information.

When I parse a site I don't want random content I want to find relevant content such as blocks of text, images, links etc. but obviously I don't want header links or footer links.

So is there anyway you can advise me to look at.. tips / tricks are also welcome :)

Regards

Answer 1

Try Simple HTML Dom Parser:

http://simplehtmldom.sourceforge.net/

// Create DOM from URL or file
$html = file_get_html('http://www.example.com/');

// Find all images 
foreach($html->find('img') as $element) 
       echo $element->src . '<br>';

// Find all links 
foreach($html->find('a') as $element) 
       echo $element->href . '<br>

Answer 2

I prefer using one of the native XML extensions, like

• DOM or
• XMLReader.

If you prefer a 3rd party lib, I'd suggest not to use SimpleHtmlDom, but a lib that actually uses DOM/libxml underneath instead of String Parsing:

• phpQuery,
• Zend_Dom,
• QueryPath,
• FluentDom or
• fDOMDocument

You can use the above for parsing HTML5, but there can be quirks due to the markup HTML5 allows. So for HTML5 you want to consider using a dedicated parser, like

• html5lib

Or use a WebService like

• YQL or
• ScraperWiki.

If you want to spend some money, have a look at

• PHP Architects Guide to Webscraping with PHP

Last and least recommended, you can - to a very limited certain degree - extract data from HTML with Regular Expressions. In general using Regular Expressions on HTML is discouraged, because HTML is not regular and the aforementioned libraries do a much better job on this. See

• Do NOT try parsing HTML with regular expressions
• Parsing Html The Cthulhu Way
• RegEx match open tags except XHTML self-contained tags

Answer 3

This is commonly referred to as screen scraping, by the way. The library I have used for this is Simple HTML Dom Parser.

Answer 4

you should also look at http://stackoverflow.com/questions/3603511/html-scraping-and-css-queries