How do you design a serial command protocol for an embedded system?

Question

I have an embedded system I'm communicating with over serial. The command structure right now is designed to be operated interactively: it displays a prompt, accepts a few commands, and displays results in a human-readable form.

I'm thinking about changing this to a more machine-usable format, so I can talk to it through a MATLAB GUI without too much trouble (right now it's hiccuping on the interactive prompts and varying message lengths, etc.).

So is there a document or standard somewhere that describes how to design a good serial command protocol for your embedded system?

Answer 1

Take a look at Firmata as an example protocol.

Answer 2

I have some preferences (and pet peeves) from writing software to control media and display devices using RS232. Depending on your hardware, some of these may not apply:

• I think it's a good idea to make your protocol more friendly for automation. If you need an interactive interface (command line or other), build it separately and have it use the automation protocol. I wouldn't worry too much about making it human readable, but it's up to you.

• Always return a response, even (especially) if you get an invalid command. Something simple like $06 for ACK and $15 for NAK. Or spell it out if you want it to be slightly more human readable.

• If you can set any value, make sure there's some way to query that same value. If you have a lot of values, it could take a while to query them all. Consider having one or just a few meta-queries that return several values at once.

• If you have information that can't be set, but is important (model no, serial no, version, copyright, etc), make sure these can be queried instead of just displaying them once on startup or reset.

• Never respond with an error for a valid command. You'd think this one would be obvious...

• Speaking of obvious, document the serial settings your hardware supports. Especially if it's going to be used by anyone other than you and you don't want them to spend the first 30 minutes trying to figure out if they can't talk to the device because of the serial port, the connections, the cable or their software. Not that I'm bitter...

• Use absolute commands instead of toggle values. For example, have separate commands for Power On and Power Off instead of sending the same command and having the power toggle on and off.

• Responses should include information on the command they are responding to. This way any program doesn't need to remember the last thing it asked for in order to deal with the response (see extra credit option below).

• If your device supports a standby mode (off, but not really off), make sure queries still work while you're in this state.

Depending on how paranoid you are about data completeness:

• Wrap your message in an envelope. The header could include a starting character, the lengeth of the message and a closing character. Just in case you get partial or malformed messages. Maybe $02 for the start and $03 for the end.

• If you're really paranoid about message integrity, include a checksum. They can be a bit of a pain, though.

For extra credit:

• If your hardware settings can be changed manually, maybe send this change out the serial port as if the user had queried it. For example, you might not want the user to be able to change the input source for a public display monitor.

I hope this helps.

Update:

I forgot something important. Before you either use this seriously and especially before you give it out to someone else, try it out on something trivial to make sure it works the way you expect it to and (more importantly) to make sure you haven't left anything out. It will take more time and effort to fix things if you find a problem in the middle of a bigger project.

This is a good rule of thumb whether you're designing a command protocol, a web service, a database schema or a class, etc.

Answer 3

FTP is an example of a protocol that works reasonably well both interactively and with automation. One key is that responses start with a code that indicates whether an automated client should pay attnetion or not. Similarly for POP3.

One nice thing about those protocols is that when your developing/debugging you can reasonbly drive the communication from a regular terminal or script the communication using regular shell scripts/batch files/whatever.

However, one thing they don't do is provide reliability - that's provided by a lower layer of the comm stack. It's something that should be considered in most embedded communication pahs.

Answer 4

Unless bandwidth or latency is a big issue, use ASCII where you can - it makes debugging much easier.

I'm fond of protocols that send a message then a clear 'message end' character (such as 'carriage return'). I don't generally find start of packet signals to be that helpful (what else is on that wire?) Using CR for message end also makes it easier to test via terminal program.

Update: Bruce pointed out (in the comments) that a start of packet character lets you find the packets slightly quicker in the case of corruption. Without the start of packet character, it would take until the end of the next packet before you knew where you were and at that point you'd be throwing out 2 packets instead of one.

Answer 5

Have you looked at Modbus (http://www.modbus.org/)? It's a fairly simple protocol which includes a checksum on each message. It also sends a response to every command, even those which don't need a "return value", so you know if your command got received correctly. The only choice you would have after choosing Modbus would be the register addresses to store your data at, and the format of any user defined functions (which the MODBUS protocol allows).

Answer 6

I like Bruce McGee's answers. Having worked with similar interfaces I can offer several other pointers:

• When returning numeric types in a data packet, please please PLEASE try to make everything the same format. Don't have single for some numbers and double for others. And don't make it arbitrary!

• Provide examples for data packets in your ICD. It's terribly frustrating to have to guess at byte order or even bit order (is MSByte first, or last? What is first and last?). Provide a diagram that shows packets vs. time (ie, 0x02 is sent earliest, then the address byte, then the message id, etc).

• Don't switch around between data formats if at all possible. I've worked in systems that use 'ASCII-encoded numbers' for some messages where you have to strip the leading '3' off of the bytes, then pull them together to make the real number. (Usually ASCII-encoding is used when you have a byte sequence you have to avoid, like 0x02, 0x04, etc. Encoded the numbers would be 0x30, 0x32, 0x30, 0x34. Use a length field if possible to avoid this, or at least do it all the time!)

• Definitely definitely definitely document the baud rate, parity, etc. If you're using RS-485 document the bus mode (2-wire? 4-wire?) or whatever settings will appear on the machine that you intend this to be used on. Give screenshots if you have to.

This interface will probably be very useful for debug. I've worked with some systems that had debug features such as:

• A system where the programmer made a list of 'logged parameters' (internal variables). You would tell the system which ones you wanted reported (up to 8) and then when you queried the system for the logged parameters it would return them in one data packet. I liked this, but depending on the complexity of the system you may or may not be able to specify them at run-time (or you could do something simple and send the system a mask that would select the ones you want returned).

• Data packets that 'break' behavior and allow parts of the system to be tested independently (ie, on a D/A put out this voltage, on a DIO port stim this byte, etc)

Good luck!

Answer 7

There are a lot of good suggestions and ideas here and notice that there are different ways to tackle it for different purposes. Having used many serial protocols both good and bad as well having made several of my own (both good and bad...) here are a few of my suggestions and comments.

• Keep it simple. I have found the greatest success with simple header based command-response "packets".

• Don't worry about human readable ASCII. It is only useful for the few hours you actually debug your protocol. After that it is restrictive to always encode data as ASCII and very inefficient if you transfer a lot of data.

• Use error checking. I prefer a 16 bit CRC as it provides orders of magnitude of protection over a checksum and is still simple and efficient over heavier algorithms like MD5 or SHA1.

• Use the same packet format for the commands as the responses.

• Use 8 bit data with no parity. Serial parity bit doesn't add anyprotection if you already use a CRCor other data integrity check and isa poor error check at best.

• So in the simplest form the packet header is the Command or Response, size of the packet, zero or more dataand the error checking code (CRC).

Answer 8

Here is a great article by Eli Benderski on serial protocol framing. Whatever the packet format you chose, be sure to use escape characters. It allows you to have such characters inside actual data and makes it really easy to re-synchronize in case of packet corruption.