Saving stack by directly writing into the send buffer below

Question

Imaging having a stack of protocols and some c/cpp code that neatly covers sending on each layer. Each send function uses the layer below to add another header until the whole message is eventually placed into a continuous global buffer on layer 0:

void SendLayer2(void * payload, unsigned int payload_length)
{
  Layer2Header header;   /* eat some stack */
  const int msg_length = sizeof(header) + payload_length;

  char msg[msg_length];  /* and have some more */
  memset(msg, 0, sizeof(msg));

  header.whatever = 42;
  memcpy(&buffer[sizeof(header)], payload, payload_length);

  SendLayer1(buffer, msg_length);
}

void SendLayer1(void * payload, unsigned int payload_length)
{
  Layer1Header header;   /* eat some stack */
  const int msg_length = sizeof(header) + payload_length;

  char msg[msg_length];  /* and have some more */
  memset(msg, 0, sizeof(msg));

  header.whatever = 42;
  memcpy(&buffer[sizeof(header)], payload, payload_length);

  SendLayer0(buffer, msg_length);
}

Now the data is moved to some global variable and actually transferred:

char globalSendBuffer[MAX_MSG_SIZE];
void SendLayer0(void * payload, unsigned int payload_length)
{
  // Some clever locking for the global buffer goes here

  memcpy(globalSendBuffer, payload, payload_length);
  SendDataViaCopper(globalSendBuffer, payload_length);
}

I'd like to reduce both the stack usage and the number of memcpy()s in this code, so I imagine something like:

void SendLayer2(void * payload, unsigned int payload_length)
{            
  Layer2Header * header = GetHeader2Pointer();
  header->whatever = 42;

  void * buffer = GetPayload2Pointer();
  memcpy(buffer, payload, payload_length);

  ...
}

My idea would be to have something at the bottom that would calculate the proper offsets for each layers header and the offset for the actual payload by continuously subtracting from MAX_MSG_SIZE and letting the upper layer code then fill in the global buffer directly from the end / right side.

Does this sound sensible? Are there alternative, perhaps more elegant approaches?

Answer 1

You may be interested in this article: "Network Buffers and Memory Management" by Alan Cox. Basically, you have the buffer and several pointers to different interesting parts of that buffer: protocol headers, data, ... Initially you reserve some space for headers by setting the data pointer to (buffer_start + max_headers_size), and each layer gets a pointer nearer to the start of the buffer.

I'm sure there must be a similar description somewhere for BSD's mbufs.

EDIT:

David Miller (Linux networking maintainer) has this article "How SKBs work"

Answer 2

This sounds like "Zero Copy." I'm no expert so search for that term and you'll find all sorts of references.