C++ byte stream

Question

Hi Everyone,
For a networked application, the way we have been transmitting dynamic data is through memcpying a struct into a (void*). This poses some problems, like when this is done to an std::string. Strings can be dynamic length, so how will the other side know when the string ends? An idea I had was to use something similiar to Java's DataOuputStream, where I could just pass whatever variables to it and it could then be put into a (void*). If this can't be done, then its cool. I just don't really like memcpying a struct. Something about it doesn't seem quite right.

Thanks,
Robbie

Answer 1

nothing wrong with memcpy on a struct - as lng as the struct is filled with fixed-size buffers. Put a dynamic variable in there and you have to serialise it differently.

If you have a struct with std::strings in there, create a stream operator and use it to format a buffer. You can then memcpy that buffer to the data transport. If you have boost, use Boost::serialize which does all this for you (that link also has links to alternative serialization libs)

Notes: the usual way to pass a variable-size buffer is to begin by sending the length, then that many bytes of data. Occasionally you see data transferred until a delimiter is received (and fields within that data are delimited themselves by another character, eg a comma).

Answer 2

I see two parts of this question: - serialization of data over a network - how to pass structures into a network stack

To serialize data over a network, you'll need a protocol. Doesn't have to be difficult; for ASCII even a cr/lf as packet end may do. If you use a framework (like MFC), it may provide serialization functions for you; in that case you need to worry about how to send this in packets. A packetization which often works well for me is :

<length><data_type>[data....][checksum]

In this case the checksum is optional, and also zero-data is possible, for instance if the signal is carried in the data_type (i.e. Ack for acklnowedgement)

If you're working on the memcpy with structures, you'll need to consider that memcpy only makes a shallow copy. A pointer is worthless once transmitted over a network; instand you should transmit the data from that pointer (i.e. the contents of your string example)

Answer 3

For sending dynamic data across the network you have the following options.

First option in the same packet. void SendData()
{
int size;
char payload[256];

Send(messageType)
Send(size);
Send(payload)
}

Second option:
void SendData()
{
char payload[256];

Send(messageType)
Send(payload)
}

Though in either situation, you will be faced with more of a design choice. In the first example you would send the message type, and the payload size and also then the payload.

The second option you have is you can send the message type and then you can send the string that has a delimiter of null terminator.

Though either option does not cover fully the problem your facing I think. Firstly, you need to determine if you're building a game what type of protocal you will be using, UDP? TCP? The second problem you will be facing is the maximum packet size. Then on top of that you need to have the framework in place so that you can calculate the optimum packet size that will not be fragmented and lost to the inter web. After that you have bandwidth control in regards to how much data you can transmitted and receive between the client and server.

For example the way that most games approach this situation is each packet is identified with the following.

MessageType
MessageSize
CRCCheckSum
MessageID
void buffer[payload]

In situation where you need to send dynamic data you would send a series of packets not just one. For example if you were to send a file accross the network the best option would to use TCP/IP because its a streaming protocal and it garnentees that the complete stream arrives safly to the other end. On the other hand UDP is a packet based protocal and is does not do any checking that all packets arrived in order or at all on the other end.

So in conclusion. 1) For dynamic data, send multiple packets but with a special flag to say more data is to arrive to complete this message. 2) Keep it simple and if your working with C++ dont assume the packet or data will contain a null terminator and check the size compared to the payload if you decide to use a null terminator.