What is the best open-source Java web crawler? I need multithreading.
I usually work with Nutch. It is based on the Lucene index and provided by Apache.