ThreadPool, QueueUserWorkItem and Deadlock on Shutdown

Question

I just implemented a thread pool like described here

Allen Bauer on thread pools

Very simple implementation, works fine, but my application no longer shuts down. Seems that two worker threads (and one other thread, I guess the queuing thread) stuck in the function

ntdll.ZwRemoveIoCompletion

I remember to have read something about IO completions in the help entry for QueueUserWorkItem (the WinAPI function used in the thread pool implementation), but I couldn't understand it properly. I used WT_EXECUTELONGFUNCTION for my worker threads since execution can take a while and I want a new worker thread created instead of waiting for the existing ones to finish. Some of the tasks assigned to the worker threads perform some I/O stuff. I tried to use WT_EXECUTEINIOTHREAD but it does not seem to help.

I should mention that the main thread waits for entry to a critical section witht the call stack being

System.Halt0, System.FinalizeUnits, Classes.Finalization, TThread.Destroy,
RtlEnterCriticalSection, RtlpWaitForCriticalSection

Any ideas what I'm doing wrong here? Thanks for your help in advance.

Answer 1

To make sure the worker threads shut down, you need to have some way of waking them up if they are waiting on the empty IO completion port. The simplest way would seem to be to post a NULL message of some kind to the port - they should then treat this as a signal to halt in an orderly fashion.

Answer 2

You must leave from the critical section before you can enter again. So the problem is inside a lock.

In some thread:

EnterCriticalSection(SomeCriticalSection);
sort code...
LeaveCriticalSection(SomeCriticalSection);

In some other thread:

EnterCriticalSection(SomeCriticalSection);
clean up code...
LeaveCriticalSection(SomeCriticalSection);

If the sort code is running in the first thread and the second thread try to run the clean up code the second thread will wait until the sort code finish and you leave the critical section. Only after leaving the critical section you can enter the same critical section. I hope this will help you narrow down the deadlock code because it is inside a critical section.

To get the completion port handle you can save it's handle when you create the completion port:

FIoCPHandle := CreateIoCompletionPort(INVALID_HANDLE_VALUE, 0 , 0, FNumberOfConcurrentThreads);

Answer 3

When using QueueUserWorkItem, as long as the worker threads have been returned to the thread pool, you should not have to do anything to shut them down. The WT_EXECUTEDEFAULT component of the thread pool queues the work items up onto an I/O completion port. This port is part of the thread pool's internal implementation and is not accessible to you.

Could you provide some more detailed call stacks for the threads that appear to be stuck? It would make this problem much easier to diagnose.