No-copy RTOS messaging

Software layering is a traditional way to handle complex software requirements. The resulting software is often referred to as a “software stack.” Common examples are protocol stacks such as TCP/IP and USB. This technique is also appropriate, and widely used for embedded software. Embedded systems normally accept data inputs and messages, process them, then output control signals and messages. Software stacks fit this requirement well.

In most embedded systems, data inputs are handled by interrupt service routines (ISRs). For input, there’s often a need to get a data block, fill it, and pass it on to a higher software layer. For output, the reverse is true: accept a data block, output the data, and release the data block. Normally in embedded systems, data blocks are stored in block pools. Since they’re called from ISRs, it’s important that the BlockGet() and BlockRel() operations be both fast and interrupt safe.

Hence, the data blocks are normally unencumbered with metadata, and thus they’re called “bare blocks.” Once filled, it’s necessary to pass an input data block up the software stack, hopefully without needing to recopy it. This is often done, in embedded systems, by passing pointers to blocks via pipes (also known as “message queues”; above the ISR level, data blocks are generally referred to as “messages.”) However, the pointer-in-pipe method lacks information needed by higher level software, such as:

  • where to return the data block, i.e. which pool, if any
  • data block size
  • where to send an acknowledgement, if needed
  • message priority
  • message owner

These are essential information to achieve the important objective of software layer independence. For example, hard-coding the message size or the return block pool in an upper layer task creates a dependency, which may lead to coding errors and which also reduces flexibility. In addition, passing unprotected block pointers is a risky practice. The pointer may be wild or not pointing to the start of the data block. The receiving task has no way to protect against these problems.

A better approach is to make a data block into a message at the ISR/task transition layer, using a MsgMake() RTOS service. This involves getting a message control block (MCB) from the MCB pool, linking it to the data block via a block pointer in the MCB, and filling in the other MCB fields. From here on, the message is passed via its handle, which is a pointer to its MCB and which can be verified to be correct. Note that the block pointer is internal to the MCB and not accessible to user code. Hence, there’s a much higher degree of confidence that it points to the start of a valid data block.

Messages are passed to tasks via “message exchanges.” Messages can wait at exchanges for processing by tasks and tasks can wait at exchanges for messages to process. Thus, exchanges can be the bridges between software layers, both up and down. It’s important to note that receiving tasks are anonymous to sending tasks and sending tasks are anonymous to receiving tasks. Tasks need only know where to get messages and where to send them. This provides software flexibility, such as easily creating server tasks, dynamic task replacement, message-token task control, message multicasting, message broadcasting, remote message assembly, and other powerful design techniques. At the end of the line, the last task calls the MsgRel() service and the block will automatically be returned to its correct pool, for reuse.

Of course, output is just the opposite: a task uses the MsgGet() service to get a message, which it fills and passes down the stack for addition filling, error coding, headers, etc. At the task/ISR transition level, the MsgUnmake() service is called to pass the block pointer, block size, and pool pointer to an ISR. It also returns the MCB to the MCB pool for reuse. The ISR does its job and returns the data block to its pool. The ISR needn’t know where the message originated; it could have come from any task at any level in the software stack. Furthermore, block pools are agnostic; they don’t care who uses their blocks. Blocks can even be from the heap or simply static; they don’t have to come from pools.

More information on this no-copy RTOS messaging technique is available in the smx User’s Guide.

Ralph Moore, President and Founder of Micro Digital, graduated with a degree in Physics from Caltech. He spent his early career in computer research, then moved into mainframe design and consulting.