Any good advice for TCP packet formatting?

Question

I am creating an application for mobile phone which sends the acceleration measurements through TCP connection to the server.

I would like to reduce the message length as much as possible but in the same time I would like to make it the current format possible to extend without a lot of pain on modifying receiver parsing mechanism.

At the beginning I send a string in a following format:

##measurementTime#AccelerationX#AccelerationY#AccelerationZ

butsoon after the implementation I added to the message some other data and I realized it will take a lot of time if I will have to modify the format frequently.

I was thinking about the XML, but it adds a lot of load which of course I would like to avoid (the measurements are sent every 100-250 ms).

Thanks!

Answer 1

I would sincerely recommend changing your packet structure to a binary format if you want to reduce size, this will also allow for easy scalability depending on how you delimit your packets.

You could do something to the extent of the following:

n byte begin+size
n byte time
2-8 byte x
2-8 byte y
2-8 byte z

Answer 2

One recommendation is sending simple key name / value pairs if you need to stick completely to an ASCII text type stream. The key name is used to describe the name of the field that each value conveys similar to original proposal:

##keyName1=value1#keyName2=value2#

Alternatively, you can send data in a binary tagged format such as this:

<tagCodeNum><lengthInBytes><tagValueAsBytes>

where tagCodeNum is perhaps a byte or word and length is a byte or word depending on your needs. The idea of this format is that the receiver can recognize fields that it understands by the code number and then can also skip tags it doesn't know how to decode. In this way, the encoding becomes extensible. If you need multiple tags grouped into logical messages, I would wrap a group of these binary coded tags in an overall message hierarchy:

<messageCodeNum><lengthInBytes><tag><tag><tag>

Where the tag above is a replication of the previous tag construction described above and length describes the byte length of all the tags combined together.

Note: If you think about this structure, not a whole lot different than an XML type structure but it is a lot more concise and constrained so that it is nearly trivial to decode.

Answer 3

thinking of something optimized:
- send only the difference with the previous value;
- from time to time send a complete frame;
- don't use xml, make your struct definition;

Answer 4

how big are each of the fields? i think i would do it in bytes, but without knowing the size of each, it is hard to really say.

Answer 5

I would like to build on InSciTek Jeff's answer a little. This is what is known more generally as tag-length-value encoding. The idea is the first code is the tag, it tells you how to interpret the value. The second code is the length which tells you how many bytes there are in the value. This is a great way to do extensible binary encoding and is used a lot in MPEG encoding.

I would add in a few more hints/requirements on a good functional binary protocol:

1. include a version number in each message that states the protocol version for the message
2. encode every multi-byte field in network byte order
3. carefully consider the size of each field
4. consider using a multi-byte encoding scheme for integers: I can't find a good reference, but the idea is that the top bit is 1 if there are more bytes and 0 if this is the last byte and the lower seven bits in each byte contains the value.
5. pack everything on to the wire as bytes and never structures

The first one is really important. It is more important that you plan for change and carefully decide how the protocol will change. Clients should never attempt to interpret a message if they don't support the version number. NEVER!

The fifth one is pretty important when it comes to having a stable implementation. I've written quite a few binary protocol implementations for embedded devices and using packed structures is the mistake that I regret the most. I have had too many defects related to forgetting to byte swap a numeric field or not packing a structure. Just write a nice library of byte packing and unpacking primitives, test it thoroughly, and use it religiously.

If you design and implement the protocol with these principles in mind, it will make your life a lot easier when you have to support and extend. The last thing that I would highly recommend is writing a dissector for something like Wireshark for your protocol. It will also make deploying, testing, and supporting things easier as well. Just make sure to completely understand the possible legal implications of writing such a dissector as well as deploying one before you consider it.

Answer 6

I was thinking about the XML, but it adds a lot of load

You can use JSON instead. It won't be as compact as a binary format but much more compact than XML. And you can easily extend the data format.