Threading Web requests handled in Main?

Question

I'm writing an application in C#, and I am creating multiple BackgroundWorker threads to grab information from webpages. Despite them being BackgroundWorkers, my GUI Form is becoming unresponsive.

When I am debugging, I pause when the program goes unresponsive, and I can see that I am in the Main Thread, and I am paused on the webpage fetching method. This method is only called from new threads, though, so I can’t figure out why I would be there in the Main Thread.

Does this make any sense? What can I do to make sure the web requests are only being handled in their respective threads?

EDIT: some code and explanation

I am processing a large list of addresses. Each thread will be processing one or more addresses. I can choose how many threads I want to create (I keep it modest :))

//in “Controller” class
public void process()
{
for (int i = 1; i <= addressList.Count && i<= numthreads; i++) 
            {
                BackgroundWorker bw = new BackgroundWorker();
                bw.DoWork += doWork;
                bw.RunWorkerAsync((object)i);
            }
}

public void doWork(object sender, DoWorkEventArgs e)
{
    //create an object that has the web fetching method, call it WorkObject
    //WorkObject keeps a reference to Controller. 
    //When it is done getting information, it will send it to Controller to print
    //generate a smaller list of addresses to work on, using e.Argument (should be 'i' from the above 'for' loop)
    WorkObject.workingMethod()
}

When WorkObject is created, it uses “i” to know what thread number it is. It will use this to get a list of web addresses to get information from (from a larger list of addresses which is shared by the main Form, the Controller, and each of the WorkObjects – each thread will process a smaller list of addresses). As it iterates over the list, it will call the “getWebInfo” method.

//in “WorkObject” class
public static WebRequest request;

public void workingMethod()
{
    //iterate over the small list of addresses. For each one, 
     getWebInfo(address)
     //process the info a bit...then
     myController.print()

//note that this isn’t a simple “for” loop, it involves event handlers and threading 
//Timers to make sure one is done before going on to the next
}

public string getWebInfo (string address)
{
    request = WebRequest.Create(address);
                WebResponse response = request.GetResponse();
                StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
                string content = reader.ReadToEnd();
                return content;
}

Answer 1

You should be performing all the web-work in your BackgroundWorker's DoWork event, are you?

Some clipped code may help us to understand what's going on.

Answer 2

In a similar situation I found I had better control by creating my own threads (no BackgroundWorker and no ThreadPool) and throttling the number of active connections. Write the IP addresses to a Queue and then pick each one off, passing it to a helper class where you call its DoSomething method on a new thread. Throttle by incrementing a counter when you launch a thread and decrement it when the thread completes. You can use callback routines from your helper class to signal your UI thread to update the interface or indicate a thread is finished.

Use your UI to vary the throttle limit and watch the Task Manager to see the effect on memory and CPU usage. You can add a maxconnection setting in your app.config to get more simultaneous connections.

Answer 3

I couldn't figure out why the background workers were causing hanging, but instead of using synchronous HTTP requests I switched to asynchronous using this guide:

http://www.developerfusion.com/code/4654/asynchronous-httpwebrequest/

Cleared up all the hanging. Thanks for the responses.