I want to post a message to a thread that is running as another process (in particular as a windows service). I've read the documentation for PostThreadMessage but there are some things unclear for me.
How do I get a handle for my service's thread?
The system only does marshalling for system messages (those in the range 0 to (WM_USER-1)). To send other messages (those >= WM_USER) to another process, you must do custom marshalling.
I want to send my own messages, so how do I do this marshalling?
PostThreadMessage is less than ideal here. Getting the thread ID is indeed a problem since your service will have to run more than one thread. One to implement the service, another to pump a message loop that's required to read the messages. CreateToolHelp32Snapshot() can enumerate the threads but you will still not know which of those two threads is the one that's pumping.
Use a named pipe instead. Call CreateNamedPipe() in your service, use message mode and give the pipe a name prefixed with "Global\" so that it is visible in the user session. The user code can connect to the pipe using the well-known pipe name. You can send whatever you want across the pipe but you will have to avoid pointers since they won't be valid in the service process. Same kind of problem as the message marshaling.
Other possibilities are a socket, very similar to a pipe but using a well-known port number instead of a name, and out-of-process COM. Using COM is an advantage if you have complex objects that need to be marshaled across the process boundary. Avoid it if you don't have COM skillz though.