Callback, specified in QueueUserAPC , does not get called

Question

In my code, I use QueueUserAPC to interrupt the main thread from his current work in order to invoke some callback first before going back to his previous work.

std::string buffer;
std::tr1::shared_ptr<void> hMainThread;
VOID CALLBACK myCallback (ULONG_PTR dwParam) {
    FILE * f = fopen("somefile", "a");
    fprintf(f, "CALLBACK WAS INVOKED!\n");
    fclose(f);
}
void AdditionalThread () {
    // download some file using synchronous wininet and store the
    // HTTP response in buffer
    QueueUserAPC(myCallback, hMainThread.get(), (ULONG_PTR)0);
}
void storeHandle () {
    HANDLE hUnsafe;
    DuplicateHandle(GetCurrentProcess(), GetCurrentThread(), 
        GetCurrentProcess(), &hUnsafe, 0, FALSE, DUPLICATE_SAME_ACCESS);
    hMainThread.reset(hUnsafe, CloseHandle);
}
void startSecondThread () {
    CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)AdditionalThread, 0, 0, NULL);
}

storeHandle and startSecondThread are exposed to a Lua interpreter which is running in the main thread along with other things. What I do now, is

1. invoke storeHandle from my Lua interpreter. DuplicateHandle returns a non-zero value and therefore succeeds.
2. invoke startSecondThread from my Lua interpreter. The additional thread gets started properly, and QueueUserAPC returns a nonzero value, stating, that all went well.
3. as far as I understood QueueUserAPC, myCallback should now get called from the main thread. However, it doesn't.

If QueueUserAPC is the correct way to accomplish my goal (==> see my other question):

• How can I get this working?

If I should some other method to interrupt the main thread:

• What other method should I use? (Note that I don't want to use pull-ing method in the main thread for this like WaitForSingleObject or polling. I want that the additional thread push-es it's data straight into the main thread, as soon as possible.)

Answer 1

From what I can understand in MSDN, the callback is not invoked until the thread enters an alertable state, and this is done by calling SleepEx, SignalObjectAndWait, WaitForSingleObjectEx, WaitForMultipleObjectsEx, or MsgWaitForMultipleObjectsEx.

So if you really don't want to do some polling, I don't think this method is adapted to your case.

Is it possible to implement a "message pump" (or rather an event listener) in your main thread and to delegate all its current work to another thread ? In this case, the main thread waits for any event that are set by the other threads.

Answer 2

Yeah, QueueUserAPC is not the solution here. Its callback will only run when the thread blocks and the programmer has explicitly allowed the wait to be alertable. That's unlikely.

I hesitate to post the solution because it is going to get you into enormous trouble. You can implement a thread interrupt with SuspendThread(), GetThreadContext(), SetThreadContext() and ResumeThread(). The key is to save the CONTEXT.Eip value on the thread's call stack and replace it with the address of the interrupt function.

The reason you cannot make this work is because you'll have horrible re-entrancy problems. There is no way you can guess at which point of execution you'll interrupt the thread. It may well be right in the middle of it mutating state, the state that you need so badly that you are contemplating doing this. There is no way to not fall into this trap, you can't block it with a mutex or whatnot. It is also extremely hard to diagnose because it will work so well for so long, then randomly fail when the interrupt timing just happens to be unlucky.

A thread must be in a well known state before it can safely run injected code. The traditional one has been mentioned many times before: when a thread is pumping a message loop is is implicitly idle and not doing anything dangerous. QueueUserAPC has the same approach, a thread explicitly signals the operating system that it is a state where the callback can be safely executed. Both by blocking (not executing dangerous code) and setting the bAlertable flag.

A thread has to explicitly signal that it is in a safe state. There is no safe push model, only pull.