webcrawler in java or with j2ee as my final year project is good?How can I improve searchengine via webcrawler? What are the other features the best crawler have?

Question

This is for my final year project. I want to design a web crawler, that crawls the pages with set of urls I have given. I want to know that How can I improve searchengine via webcrawler? I choosed java as my developing language.Is that good? What are the other features the webcrawlers have? What are the other ways it can help us? What are the other applications it have other than in searchengine?

Answer 1

If your web crawler ain't distributed, it ain't a webcrawler.

Distributed Web Crawling