Good method to pass messages between embedded RTOS tasks (but can handle message timeouts gracefully)

Question

I'm working with an embedded RTOS (CMX), but I think this applies to any embedded RTOS. I want to pass messages between various tasks. The problem is that one task sometimes 'locks' every other task out for a long period of time (several seconds).

Since I no longer wait for the message to be ACK'ed after ~100 ms or so, if I send a mailbox message during this time, the task that sent the message is no longer waiting for it reply, but the receiving task will get the message and try to act on it. The problem is that the receiving task has a pointer to the message, but since the sending task has moved on, the pointer is no longer pointing to the message which can cause huge problems.

I have no method of removing messages once they are in the queue. How can I handle this error gracefully?

Answer 1

This question actually covers several different issues / points.

First of all, I'm wondering why one task hogs the CPU for seconds at a time sometimes. Generally this is an indication of a design problem. But I don't know your system, and it could be that there is a reasonable explanation, so I won't go down that rabbit hole.

So from your description, you are enqueueing pointers to messages, not copies of messages. Nothing inherently wrong with that. But you can encounter exactly the problem you describe.

There are at least 2 solutions to this problem. Without knowing more, I cannot say which of these might be better.

The first approach would be to pass a copy of the message, instead of a pointer to it. For example, VxWorks msg queues (not CMX queues obviously) have you enqueue a copy of the message. I don't know if CMX supports such a model, and I don't know if you have the bandwidth / memory to support such an approach. Generally I avoid this approach when I can, but it has its place sometimes.

The second approach, which I use whenever I can in such a situation, is to have the sender allocate a message buffer (usually from my own msg/buffer pools, usually a linked-list of fixed size memory blocks - but that is an implementation detail - see this description of "memory pools" for an illustration of what I'm talking about). Anyway -- after the allocation, the sender fills in the message data, enqueues a pointer to the message, and releases control (ownership) of the memory block (i.e., the message). The receiver is now responsible for freeing/returning the memory after reading the message.

There are other issues that could be raised in this question, for example what if the sender "broadcasts" the msg to more than one receiver? How do the receivers coordinate/communicate so that only the last reader frees the memory (garbage collection)? But hopefully from what you asked, the 2nd solution will work for you.