Guidelines for designing forward compatible communication protocols?

Question

I'm working on a communication protocol between embedded devices. The protocol will definitely need new commands and fields in the future. What do I need to do to make sure I'm not painting myself into a corner?

Answer 1

The question is a little too general for a clear answer. There are many aspects an embedded system may need to communicate like;

How many peers will it need to communicate with? How much data does it need to communicate? How tightly synchronized do the systems need to be? What is the physical media for the protocol and what are the bandwidth limitations, and error susceptibility considerations?

All of these requirements and resource limitations will certainly constrain the system and then you can start to figure out what the protocol will need. Once you know these issues you can then project how some the requirements may change/expand in the future. From there you can design the protocol to accommodate(or not) the worst case use cases.

Answer 2

This is a wide open question. Here are some random thoughts about it:

1. Leave spares.
2. Use a very basic header with a "number of bytes to follow" field.
3. If there are enumerated message types, make sure the type field can accomodate growth.
4. If you use bitflags, leave spares.
5. Possibly include a "raw data" message, which can be used to wrap any protocol future generations think up.

In summary, Leave spares.

Answer 3

I would use HDLC. I have had good luck with it in the past. I would for a point to point serial just use the Asynchronous framing and forget about all of the other control stuff as it would probably be overkill.

In addition to using HDLC for the framing of the packet. I format my packet like the following. This is how options are passed using 802.11

U8 cmd;
U8 len;
u8 payload[len];

The total size of each command packet is len +2

You then define commands like

#define TRIGGER_SENSOR 0x01
#define SENSOR_RESPONSE 0x02

The other advantage is that you can add new commands and if you design your parser correctly to ignore undefined commands then you will have some backwards compatibility.

So putting it all together the packet would look like the following.

 // total packet length minus flags len+4
 U8 sflag;   //0x7e start of packet end of packet flag from HDLC
 U8 cmd;     //tells the other side what to do.
 U8 len;     // payload length
 U8 payload[len];  // could be zero len
 U16 crc;
 U8 eflag;   //end of frame flag

The system will then monitor the serial stream for the flag 0x7e and when it is there you check the length to see if it is pklen >= 4 and pklen=len+4 and that the crc is valid. Note do not rely on just crc for small packets you will get a lot of false positives also check length. If the length or crc does not match just reset the length and crc and start with decoding the new frame. If it is a match then copy the packet to a new buffer and pass it to your command processing function. Always reset length and crc when a flag is received.

For your command processing function grab the cmd and len and then use a switch to handle each type of command. I also require that a certain events send a response so the system behaves like a remote procedure call that is event driven.

So for example the sensor device can have a timer or respond to a command to take a reading. It then would format a packet and send it to the PC and the PC would respond that it received the packet. If not then the sensor device could resend on a timeout.

Also when you are doing a network transfer you should design it as a network stack like the OSI modle. The HDLC is the data link layer and the RPC and command handling is the Application Layer.

Answer 4

If at all possible, allow a human at one end of a cable to figure out what is at the other end of the cable. Ideally, a human could hook up a dumb terminal and hit the keyboard three times (Enter Question-mark Enter), then a long, detailed message would come back describing what kind of machine it is, what is its model number, the name and phone number and web site of the organization that built it, the "official" protocol version number, and the unofficial build time:

__DATE__ ": " __TIME__

Also send the same detailed message every time the machine boots up.

If at all possible, try to design your protocol so that a human being with a dumb terminal can talk to your device. The HTTP is one such human-readable protocol, and I suspect this is one of the reasons for its popularity. Human-readable implies, among other things:

• Limit yourself to characters that a human can read and type. Avoid special control characters. Take advantage of the power of plain text.
• Always send CR+LF at the end of each packet (as mandated by many Internet protocols).
• Accept characters at any rate, from maximum-speed file upload from a PC to a non-touch-typing human slowly pecking at a keyboard.

You might also want to glance over the list of common protocols for embedded systems. Perhaps one already meets your requirements? Is there any reason to use something more difficult to decode than the standard Netstring format?