What is a good Java web crawler library?

Question

Hi,

I am about to develop a crawler in Java but don't feel like reinventing the wheel. A quick Google search gives a whole bunch of Java libraries to build a web crawler. Besides that Nutch is of course a very robust package but seems a bit too advanced for my needs. I only need to crawl a handful websites a week containing a couple of 1000 pages each.

Which open source Java library would you recommend considering:

• speed
• multithreading (or even distributed)
• extending it with new functionality
• active maintained
• and documentation?

Answer 1

Try solr. It has all the spice you just need.

Answer 2

After giving the link you provided a quick read it looks Java Web Crawler looks the best for what you want.

Answer 3

Have a look at Niocchi.

Answer 4

crawler4j is a simple java crawler that can be configured in a few minutes and can easily handle a few million web pages.