tags:

views:

27

answers:

0

Hi, I am writing custom search task using nutch for intranet crawl. I am using Hadoop for it. I want to spawn the task across multiple hadoop slaves by dividing the seed urls evenly. I guess this job is taken care by the partioner.

I see the default implementation of Nutch UrlPartitioner partitions url by Host, Domain or IP. I want to override that behavior and simply divide the seeds equally based on number of maxthreads I pass on command line.

Could I do that with simple config changes other than re-writing the Partioner?

EDIT
Custom search task is being written by re-writing Crawl.java

Thanks Nayn