Class method as winAPI callback

Question

Is it feasible to set the winAPI message callback function as a method of a class. If so, how would this be best implemented? I wonder if it is even possible.

Sorry for the short question, hopefully you will be able to provide useful responses.

Thanks in advance :).

Answer 1

You cannot use non-static member functions for C callbacks.

However, usually C callbacks have a user data pointer that's routed to the callback. This can be explored to do what you want with the help of a static member functions:

// Beware, brain-compiled code ahead!

typedef void (*callback)(int blah, void* user_data);

void some_func(callback cb, void* user_data);

class my_class {
public:
  // ...
  void call_some_func()
  {
     some_func(&callback_,this);
  }
private:
  void callback(int blah)
  {
    std::cout << blah << '\n';
  }
  static void callback_(int blah, void* user_data)
  {
    my_class* that = static_cast<my_class*>(user_data);
    that->callback(blah);
  }
};

Answer 2

Provided that it has the right signature, and the right calling convention, you can pass static class methods as callback functions for various Win32 APIs. You can't use non-static methods, as they expect one more implicit parameter (the this pointer), which the Win32 APi can't give to them.

Answer 3

You can set an appropriate static member function of a class as a callback function from Windows APIs.

Answer 4

If you use MSVC, and target x86-32 (not x86-64), you may use __stdcall convention for member function. (__cdecl works too)

With __stdcall this will be passed as first parameter, so you can write

typedef void(__stdcall *callback_t)(void* arg);
void set_callback(callback_t callback, void* callbackParameter);

struct Foo
{
   int x;
   void __stdcall someCallback() { this->x = 1; }
};

but

Foo foo;
set_callback((callback_t)&Foo::someCallback, this);

Will not work: you can't directly cast member function pointer to pointer by standard. You should use a workaround to make this working:

First, assert that &Foo::someCallback has the same size as void*

static_assert(sizeof(&Foo::someCallback) == sizeof(void*), "");
// or assert()

This assert may fail, if there may be multiple inheritance, and someCallback is virtual. To disallow multiple inheritance, use __single_inheritance keyword:

struct __single_inheritance Foo { /* ....*/ };

Second, you should cast &Foo::someCallback to callback_t by using

union
{
    void(__stdcall Foo::*src)();
    callback_t dst;    
} u = {&Foo::someCallback};
set_callback(u.dst, this);

or

void(__stdcall Foo::*temp)() = &Foo::someCallback;
set_callback(*(callback_t*)&temp, this);

It works only if you can pass this to callback as first parameter.

If you can't, you can try to dynamically generate callback stubs in assembly :) Allocate an executable memory and write there

B9 XX XX XX XX mov ecx, <this>
68 XX XX XX XX push <function member address>
C3 ret

It will be callback stub, which will convert __stdcall to __thiscall.