Thread Communication

Question

Is there any tool available for tracing communication among threads; 1. running in a single process 2. running in different processes (IPC)

Answer 1

I am presuming you need to trace this for debugging. Under normal circumstances it's hard to do this, without custom written code. For a similar problem that I faced, I had a per-processor tracing buffer, which used to record briefly the time and interesting operation that was performed by the running thread. The log was a circular trace which used to store data like this:

struct trace_data {
  int op;
  void *data;
  struct time t;
  union {
    struct {
      int op1_field1;
      int op1_field2;
    } d1;
    struct {
      int op2_field1;
      int op2_field2;
    } d2
  } u;
}

The trace log was an array of these structures of length 1024, one for each processor. Each thread used to trace operations, as well as time to determine causality of events. The fields which were used to store data in the "union" depended upon the operation being done. The "data" pointer's meaning depended upon the "op" as well. When the program used to crash, I'd open the core in gdb and I had a gdb script which would go through the logs in each processor and print out the ops and their corresponding data, to find out the history of events.

For different processes you could do such logging to a file instead - one per process. This example is in C, but you can do this in whatever language you want to use, as long as you can figure out the CPU id on which the thread is running currently.

Answer 2

You might be looking for something like the Intel Thread Checker as long as you're using pthreads in (1).

For communication between different processes (2), you can use Aspect-Oriented Programming (AOP) if you have the source code, or write your own wrapper for the IPC functions and LD_PRELOAD it.

Edit: Whoops, you said tracing, not checking.