C# Wait until all threads terminated in ThreadPool

Question

I have one main thread, and many other background threads.

The main usage of those background threads is to query data (many queries from the web, which is why I create multiple threads: to avoid the lagging the user interface).

When it comes to exporting the data in the main thread (the user interface), I need to wait until all the other threads are finished.

My code is:

//...code to open save file dialog...

//this loop is to wait for all the threads finish their query
//QueryThread.threadCount is the count of the background threads
while (QueryThread.threadCount != 0)
{
    Thread.CurrentThread.Join(1000);
    Console.WriteLine(QueryThread.threadCount);
}

//...code to export data...

If I comment the while loop out, the program will run smoothly, but some of my exported data will have possibility of showing some "unwanted" material since some of the background threads haven't finished their work.

However, the above while loop is infinite, the threadCount never changes, which means during the "Join()" method, no background thread is running.

Why are the background threads blocked and how can I solve the problem?

Thanks a lot!

Answer 1

Your implementation is wrong, and you shouldn't use Join as a synchronization primitive between your threads.

What you should do is implement the producer-consumer pattern. This will allow you to have the threads waiting to do work, and then come alive to do that work when you place it in the queue.

However, the change that I would make is that from the UI thread, don't add the data directly to the queue that the producer and consumer share. Rather, make a copy of that data and then put that in the queue.

For more information on how to implement the producer-consumer pattern in .NET, I suggest you read the MSDN documentation titled "How to: Synchronize a Producer and a Consumer Thread (C# Programming Guide)"

Answer 2

You are calling the Join method on the current thread which doesn't make much sense. You should call it on your worker threads:

foreach (Thread thread in workerThreads)
{
    thread.Join(1000);
}

Unfortunately this kind of defeats the purpose of using threads because it will block the calling until all other threads are finished.

The RunWorkerCompleted event of a BackgroundWorker could be used to notify the completion of a background task and perform updates on the form.

Answer 3

I think you want to look into signalling. Have signals (ManualResetEvent/AutoResetEvent) for your threads. Set() the associated signal handle in the worker thread when it´s done. In the main thread do a `WaitAll(signal1, signal2,signal3)´ to wait for the completion of your worker threads.

Hope this helps,

Answer 4

Sorry, I still can't figure out how to solve my problem.

Maybe I should make it specific:

"In the thread of the Main Form (which has a [STAThread] attribute), how to wait until all the thread in the ThreadPool class are done?"

I think the above answer of signal is quite close.

Btw, I have hurdreds of thread in the ThreadPool and the WaitAll() method only allow 64 WaitHandle.....

Thanks a lot

Answer 5

I solved the problem by changing my approach to Producer-Consumer model.

Thanks all. Please look at this link (provided by casperOne above), but be caution don't follow the implementation of microsoft....

Go here instead will give you a better answer.

Of course I've made some changes, the type of the Queue is Delegate in my case.

public static class QueryThread
{
    private static SyncEvents _syncEvents = new SyncEvents();
    private static Queue<Delegate> _queryQueue = new Queue<Delegate>();

    static Producer queryProducer;
    static Consumer queryConsumer;

    public static void init()
    {
        queryProducer = new Producer(_queryQueue, _syncEvents);
        queryConsumer = new Consumer(_queryQueue, _syncEvents);

        Thread producerThread = new Thread(queryProducer.ThreadRun);
        Thread consumerThread = new Thread(queryConsumer.ThreadRun);

        producerThread.IsBackground = true;
        consumerThread.IsBackground = true;

        producerThread.Start();
        consumerThread.Start();
    }

    public static void Enqueue(Delegate item)
    {
        queryQueue.Enqueue(item);
    }
}

When a query is needed in the main thread, enqueue a delegate points to the function that makes query by calling the Enqueue(Delegate item). This add a delegate to the "private" queue of the Producer.

The producer will add the items in its own queue to the shared queue on the suitable occasion (like generating random number and put it into the shared queue in the msdn example).

The consumer dequeue the delegates and run them.

Thanks all for helping. =]