IO completion port key confusion

Question

I'm writing an IO completion port based server (source code here) using the Windows DLL API in Python using the ctypes module. But this is a pretty direct usage of the API and this question is directed at those who have a knowledge of IOCP, not Python.

As I understand the documentation for CreateIoCompletionPort, you specify your "user defined" completion key when you call this function with a file handle (in my case a socket) you are associating with the created IOCP. When you get around to calling GetQueuedCompletionStatus, you get a completion key value along with a pointer to an overlapped object. The completion key should identify what overlapped object and request has completed.

However, let's say I pass in 100 as the completion key in my CreateIoCompletionPort call with an overlapped object. When the same overlapped object has its IO completed and it arrives back through GetQueuedCompletionStatus, the completion key that accompanies it is much larger and bares no resemblance to the original value of 100.

Am I misunderstanding how the completion key works, or must I be doing it wrong in the source code I linked above?

Answer 1

GetQueuedCompletionStatus returns two things, an OVERLAPPED structure and a completion key. The completion key represents per-device information, and the OVERLAPPED structure represents per-call information. The completion key should match what was given in the call to CreateIoCompletionPort. Typically, you would use a pointer to a structure containing information about the connection as the completion key.

It looks like you're not doing anything with completionKey as returned by GetQueuedCompletionStatus.

I'm guessing you want:

if completionKey != acceptKey:
    Cleanup()
    ...

edit:

Does Python somehow know that the OVERLAPPED structure created in CreateAcceptSocket is being used asynchronously by the Win32 API and prevent it from being GC'd?

Answer 2

The problem is in how I am passing the completion keys. The completion key argument is a pointer, yet it passes back the pointer not the value pointed to - a little confusing to me at least.

Also the completion key passed for the accepted connection overlapped packet is that of the listening socket - not the accepted socket.

Answer 3

What I've found in everyday practice is that it is best to simply focus on the OVERLAPPED result, as that will be unchanged. One way that you can use it effectively is to have something like the following:

struct CompletionHandler
{
    OVERLAPPED dummy_ovl;
    /* Stuff that actually means something to you here */
};

When you post something to the IOCP (whether via I/O call or just a post via Win32 API), you first create a CompletionHandler object that you will use to track the call, and cast the address of that object to OVERLAPPED*.

CompletionHander my_handler;
// Fill in whatever you need to in my_handler
// Don't forget to keep the original my_handler!

// I/O call goes here, and for OVERLAPPED* give: (OVERLAPPED*)&my_handler

This way, when you get the OVERLAPPED result all you have to do is cast it back to CompletionHandler and voila! You have the original context of your call.

OVERLAPPED* from_queued_completion_status;
// Actually get a value into from_queued_completion_status

CompletionHandler* handler_for_this_completion = (CompletionHandler*)from_queued_completion_status;
// Have fun!

For more details in a real-world setting, check out Boost's implementation of ASIO for Windows (ver 1.42 header here). There are some details like validation the OVERLAPPED pointer you get from GetQueuedCompletionStatus, but again, see the link for a good way to implement.

Answer 4

Completion key is not a pointer - it is a number of type ULONG_PTR, which just means "an integer the size of a pointer" so it is 32-bits on x86 and 64-bits on x64. The typename is confusing, but when win32 typenames refer to a pointer they do it by having a P at the front of the name, not the end.