Hi, I am writing custom search task using nutch for intranet crawl. I am using Hadoop for it. I want to spawn the task across multiple hadoop slaves by dividing the seed urls evenly. I guess this job is taken care by the partioner.
I see the default implementation of Nutch UrlPartitioner partitions url by Host, Domain or IP. I want to override that behavior and simply divide the seeds equally based on number of maxthreads I pass on command line.
Could I do that with simple config changes other than re-writing the Partioner?
EDIT
Custom search task is being written by re-writing Crawl.java
Thanks Nayn