What are the best prebuilt libraries for doing Web Crawling in Python

Question

I need to crawl and store locally for future analysis the contents of a finite list of websites. I basically want to slurp in all pages and follow all internal links to get the entire publicly available site.

Are there existing free libraries to get me there? I've seen Chilkat, but it's for pay. I'm just looking for baseline functionality here. Thoughts? Suggestions?

---

Exact Duplicate: Anyone know of a good python based web crawler that I could use?

Answer 1

Do you really need a library? I strongly recommend Heritrix as a great general purpose crawler that will preserve the whole webpage (as opposed to the more common crawlers that store only part of the text). It's a bit rough around the edges, but works great.

That said, you could try the HarvestMan http://www.harvestmanontheweb.com/

Answer 2

Use Scrapy.

It is a twisted-based web crawler framework. Still under heavy development but it works already. Has many goodies:

• Built-in support for parsing HTML, XML, CSV, and Javascript
• A media pipeline for scraping items with images (or any other media) and download the image files as well
• Support for extending Scrapy by plugging your own functionality using middlewares, extensions, and pipelines
• Wide range of built-in middlewares and extensions for handling of compression, cache, cookies, authentication, user-agent spoofing, robots.txt handling, statistics, crawl depth restriction, etc
• Interactive scraping shell console, very useful for developing and debugging
• Web management console for monitoring and controlling your bot
• Telnet console for low-level access to the Scrapy process

Example code to extract information about all torrent files added today in the mininova torrent site, by using a XPath selector on the HTML returned:

class Torrent(ScrapedItem):
    pass

class MininovaSpider(CrawlSpider):
    domain_name = 'mininova.org'
    start_urls = ['http://www.mininova.org/today']
    rules = [Rule(RegexLinkExtractor(allow=['/tor/\d+']), 'parse_torrent')]

    def parse_torrent(self, response):
        x = HtmlXPathSelector(response)
        torrent = Torrent()

        torrent.url = response.url
        torrent.name = x.x("//h1/text()").extract()
        torrent.description = x.x("//div[@id='description']").extract()
        torrent.size = x.x("//div[@id='info-left']/p[2]/text()[2]").extract()
        return [torrent]