Does any open, simply extendible web crawler exists?

Question

I search for a web crawler solution which can is mature enough and can be simply extended. I am interested in the following features... or possibility to extend the crawler to meet them:

• partly just to read the feeds of several sites
• to scrap the content of these sites
• if the site has an archive I would like to crawl and index it as well
• the crawler should be capable to explore part of the Web for me and it should be able to decide which sites matches the given criteria
• should be able to notify me, if things possibly matching my interest were found
• the crawler should not kill the servers by attacking it by too many requests, it should be smart doing crawling
• the crawler should be robust against freak sites and servers

Those things above can be done one by one without any big effort, but I am interested in any solution which provide a customisable, extendible crawler. I heard of Apache Nutch, but very unsure about the project so far. Do you have experiences with it? Can you recommend alternatives?

Answer 1

I heartily recommend heritrix. It is VERY flexible and I'd argue is the most battle tested freely available open source crawler, as it's the one the Internet Archive uses.

Answer 2

You should be able to find something that fits your needs here.

Answer 3

A quick search on GitHub threw up Anemone, a web spider framework which seems to fit your requirements - particularly extensiblility. Written in Ruby.
Hope it goes well!

Answer 4

I've used Nutch extensively, when I was building the open source project index for my Krugle startup. It's hard to customize, being a fairly monolithic design. There is a plug-in architecture, but the interaction between plug-ins and the system is tricky and fragile.

As a result of that experience, and needing something with more flexibility, I started the Bixo project - a web mining toolkit. http://openbixo.org.

Whether it's right for you depends on the weighting of factors such as:

1. How much flexibility you need (+)
2. How mature it should be (-)
3. Whether you need the ability to scale (+)
4. If you're comfortable with Java/Hadoop (+)