Avoid copying of data between user and kernel space and vice-versa

Question

Hello, I am developing a active messaging protocol for parallel computation that replaces TCP/IP. My goal is to decrease the latency of a packet. Since the environment is a LAN, i can replace TCP/IP with simpler protocol to reduce the packet latency. I am not writing any device driver and i am just trying to replace the TCP/IP stack with something simpler. Now I wanted to avoid copying of a packet's data from user space to kernel space and vice-versa. I heard of the mmap(). Is it the best way to do this? If yes, it will be nice if you can give links to some examples. I am a linux newbie and i really appreciate your help.. Thank you...

Thanks, Bala

Answer 1

You should use UDP, that is already pretty fast. At least it was fast enough for W32/SQLSlammer to spread through the whole internet.

About your initial question, see the (vm)splice and tee Linux system calls.

From the manpage:

The three system calls splice(2), vmsplice(2), and tee(2)), provide userspace programs with full control over an arbitrary kernel buffer, implemented within the kernel using the same type of buffer that is used for a pipe. In overview, these system calls perform the following tasks:

splice(2)

  moves data from the buffer to an arbitrary file descriptor, or vice

versa, or from one buffer to another.

tee(2)

  "copies" the data from one buffer to another.

vmsplice(2)

  "copies" data from user space into the buffer.

Though we talk of copying, actual copies are generally avoided. The kernel does this by implementing a pipe buffer as a set of reference-counted pointers to pages of kernel memory. The kernel creates "copies" of pages in a buffer by creating new pointers (for the output buffer) referring to the pages, and increasing the reference counts for the pages: only pointers are copied, not the pages of the buffer.

Answer 2

Since the environment is a LAN, i can replace TCP/IP with simpler protocol to reduce the packet latency

Generally, even in LAN UDP packets tend to be lost, also they will be lost if client do not have enough time to consume it...

SO no, do not replace TCP with something else (UDP). Because if you do need reliable delivery TCP would be the fastest (because everything connected to acknowledgments and retransmission is done in kernel space).

Generally in normal case there is no latency drawbacks using TCP (of course do not forget TCP_NODELAY option)

About sharing the memory. Actually all memory you allocate is created with mmap. So the kernel will need to copy it somehow in any case when it creates a packet from driver.

If you are talking about reducing copying it is usually done for files/sockets and sendfile() used that indeed prevents copying data between kernel and user. But I assume you do not need to send files.