web crawlering help required

Question

hi i am completing a little hobby project of mine to create a small scale search engine.

i was wondering if any one knows of a decent robust opensource web crawler that they have used? it should be easy for a noob to setup and use.

thank you for not googling web crawlers and pasting a list .

Answer 1

crawler4j is a pretty decent crawler, multi-threaded and easy to configure and use. It's written in Java.

You can find a list of open-source crawlers in this wikipedia page.