Overlapped I/O: How to wake a thread on a completion port event or a normal event?

Question

I want to use a thread pool to both initiate/cancel overlapped read operations -- using ReadFile() and CancelIo() respectively -- as well as handling any completion port events when read operations complete.

1. Any thread can initiate a read operation
2. Any thread can handle a read-complete event
3. Only the thread that initiated a read may cancel it (this is a CancelIo() limitation)

I'm not sure how to implement this. One normally uses GetQueuedCompletionStatus() to wait on completion port events and WaitOnSingleObject() to wait on normal events but it's not clear how to mix the two. If PostQueuedCompletionStatus() would let me specify a specific thread to wake up I'd be set. Any ideas?

UPDATE: The solution must run on Windows XP. Unfortunately this rules out using CancelIoEx() or GetQueuedCompletionStatusEx().

Answer 1

1 and 2 are easy, just use the IO Completion port.

But, as you have found 3 requires (prior to Windows V6¹) the same thread.

If using Windows >= V6, GetQueuedCompletionStatusEx includes an alterable option which will cause it to return is an APC is performed on the thread. So use QueueUserAPC to queue a no-op APC¹ when you need that specific thread to do some other work. You will of course need some thread safe queue to provide the interrupted thread with instructions of what to cancel.

If earlier version compatibility is required then things get more difficult. Possibilities:

• Use the timeout parameter of GetQueuedCompletionStatus](http://msdn.microsoft.com/library/aa364986) to return regularly to check for cancellations.

• Or, possibly more practically, divide the thread pool into two groups. Threads that initiate and cancel IO. The rest of the time these threads spend waiting to be signalled to perform one of these actions. The other part of the pool waits on IO completions with GetQueuedCompletionStatus.

Neither of these is as nice, but that's always the problem with older versions: they lack functionality.

¹ Use a no-op APC rather than doing work in the APC so the limitations on what can be done in an APC and its inherant issues with concurrency are bypassed. (Since an APC is executed on a thread, any locks that thread hold are held in the APC, any state protected will be arbitrarily inconsistent.)