ansaurus

Question

Screenscraping the ugliest HTML you've ever seen in your life

Answer 1

A:

Maybe you'd have better luck scraping the results you need using regular expressions instead of parsing it as XML.

Chris 2009-04-09 01:58:14

Answer 2

+3 A:

There's a few tricks you can use to clean up highly-predictable structures like tables. Before running HTML tidy, you can use Regex or something to search for <tr>'s and <td>'s which are followed by another <tr> or <td>, and insert the corresponding closer immediately before it. There's some added trickery for accommodating tables inside of a <td> but nothing that's impossible to handle. Just start by locating the innermost structure and moving outward from there.

The real puzzle is things like unclosed <div>'s and <p>'s, which can be much harder to match up with their corresponding (or lacking) closers.

Rex M 2009-04-09 01:59:20

Answer 3

+2 A:

If you are open to other languages such as Python, Beautiful Soup is great at reconstructing poorly written HTML. I just tried running your HTML through the following snippet, and it's now quite readable.

#!/usr/bin/env python

from BeautifulSoup import BeautifulSoup

html = "long string of html"
soup = BeautifulSoup(html)
print soup.prettify()

Dave K 2009-04-09 02:02:22

Yes - for the record I'm not necessarily dependent upon PHP to get this done.

Andy Baird 2009-04-09 02:04:15

Definitely check out Beautiful Soup then. It's by far the best screen scraper I've used. There's also a Ruby version available, in case you prefer it to Python.

Dave K 2009-04-09 02:11:16

Answer 4

+2 A:

If you are looking for data, I would just remove all html and handle it as line-by-line raw input. You can use the strip_tags function.

$clean = strip_tags($input);

// example: <p>Test paragraph.</p> <a href="#fragment">Other text</a>
// returns: Test paragraph. Other text

St. John Johnson 2009-04-09 04:19:54

Answer 5

A:

I used xpath with Python's lxml library to parse IMDB Top 250 page. View the source for yourself to see how bad it is.

The following code parses a saved IMDB Top 250 page (top250.html) and stores the extracted information in a sqlite database (top250.db)

import sqlite3
from lxml import html

tree = html.parse('top250.html')

class TopMovie(object):
    base_xpath = "/html/body/div/div[2]/layer/div[3]/table/tr/td[3]/div/table/tr/td/table/tr[%d]"

    def __init__(self, num):
        self.rank = num
        self.xpath = self.base_xpath % (self.rank + 1)

    def rating(self):
        return tree.xpath(self.xpath + '/td[2]/font')[0].text

    def link(self):
        return tree.xpath(self.xpath + '/td[3]/font/a')[0].values()[0]

    def title(self):
        return tree.xpath(self.xpath + '/td[3]/font')[0].text_content()

    def votes(self):
        return tree.xpath(self.xpath + '/td[4]/font')[0].text


def main():
    conn = sqlite3.connect('top250.db')
    conn.execute("""DROP TABLE IF EXISTS movies""")
    conn.execute("""
        CREATE TABLE movies (
            id INTEGER PRIMARY KEY,
            title TEXT,
            link TEXT,
            rating TEXT,
            votes INTEGER
        )""")

    for n in xrange(1, 251):
        m = TopMovie(n)
        query = r'INSERT INTO movies VALUES (%d, "%s", "%s", "%s", "%s")' \
            % (n, m.title(), m.link(), m.rating(), m.votes().replace(',', ''))
        conn.execute(query)

    conn.commit()
    conn.close()


if __name__ == "__main__":
    main()

Imran 2009-04-09 04:37:34

ansaurus

tags:

views:

answers:

Screenscraping the ugliest HTML you've ever seen in your life

related questions