How to handle thread exceptions in main thread, block all workers and notify them upon finishing?

Question

I have a windows service doing some repetitive work (wcf calls) after logging programmaticaly at a web server. All calls are invoked through various threads running dependently in parallel, but if for example, the connectivity is lost, or the iis recycles, hence the inproc session management directs me to re-login, I want to handle this exception, halt all other threads, propagate the exception up to main thread, retry to login and upon success to signal all threads to restart.

My main application is something like the following. I login, I start two workers and the exception handling mechanism in main thread is pending:

        IFileManagerService fileManagerService = new FileManagerService();
        IJobManagerService  jobManagerService  = new JobManagerService();

        WindowsServiceSettings.AuthenticationService.Login(WindowsServiceSettings.Credentials.Split(':')[0], WindowsServiceSettings.Credentials.Split(':')[1]);

        WriteToLogFile("Service Started at " + DateTime.Now);

        fileManagerService.Start();
        jobManagerService.Start();

Can I implement this via wait handles and how am I going to?

Thank you very much!

Answer 1

First, you need a way to propogate exceptions across threads. By far the easiest way to do this is to make your child threads into Task objects. You can then give these tasks a continuation to detect if any of them error out.

The second thing you'll need is a way to cancel the other threads. Use cooperative cancellation with the same CancellationToken. When the task continuation detects an error, it should cancel all the other threads (and wait for them to complete).

From there, it's a simple matter of having the task continuation re-login and then restart the tasks (including registering itself as a task continuation for the new tasks).

Answer 2

For a .Net 3.5 project, I'd use a callback method from a worker thread to signal the main thread that there's a problem. Have each worker invoke a separate "heartbeat" callback method that periodically checks with the main thread for permission to continue; a worker could be called from various places in its processing loop.

If the main thread receives an error callback, then it can set the heartbeat method to return a stop indication, letting each worker thread safely stop itself. When the worker stops, have it calls a termination callback that passes the thread's ManagedThreadId, which the main removes from a collection of active threads. Once the collection count is zero then all of the workers are done. This avoids having to rely on Thread.Abort().

If needed, you could use the heartbeat to reset a timer for each active thread, which could alert the main thread if a worker hangs. The heartbeat could also be used to update the main thread with worker progress.