I am running a crawler and tried to implement a multithreaded crawler. I created multiple threads to take a url from a list and go crawl it. What I would like it to do is either have all the threads fetch a url concurrently or for each thread to fetch one and then parse the html. What I have so far is not working and previous posts have not received much help. Please can someone look at my code:
startwatch.Start()
For Each link In completeList
'ThreadPool.QueueUserWorkItem(New System.Threading.WaitCallback _
' (AddressOf processUrl), link)
Dim thread = New Thread(AddressOf processUrl)
thread.Start(link)
numThread = numThread + 1
If numThread < 10 Then
ThreadList.Add(thread)
End If
Next
For Each thread In ThreadList
thread.Join()
Next
startwatch.Stop()
elapsedTime = startwatch.ElapsedMilliseconds
Trying to debug and follow the steps is confusing and what I did was lock out the function that writes to a file, but the links that come back from the parser have the error 'Property Evaluation Failed'.
Can someone please help me with this? I would even submit all my code in private if someone can help?