tags:

views:

244

answers:

0

I am running a crawler and tried to implement a multithreaded crawler. I created multiple threads to take a url from a list and go crawl it. What I would like it to do is either have all the threads fetch a url concurrently or for each thread to fetch one and then parse the html. What I have so far is not working and previous posts have not received much help. Please can someone look at my code:

 startwatch.Start()

 For Each link In completeList
     'ThreadPool.QueueUserWorkItem(New System.Threading.WaitCallback _
     '                   (AddressOf processUrl), link)
     Dim thread = New Thread(AddressOf processUrl)
     thread.Start(link)
     numThread = numThread + 1

     If numThread < 10 Then
         ThreadList.Add(thread)
     End If
 Next

 For Each thread In ThreadList
     thread.Join()
 Next

 startwatch.Stop()
 elapsedTime = startwatch.ElapsedMilliseconds

Trying to debug and follow the steps is confusing and what I did was lock out the function that writes to a file, but the links that come back from the parser have the error 'Property Evaluation Failed'.

Can someone please help me with this? I would even submit all my code in private if someone can help?