one two-directed tcp socket OR two one-directed? (linux, high volume, low latency)

Question

Hello

I need to send (interchange) a high volume of data periodically with the lowest possible latency between 2 machines. The network is rather fast (e.g. 1Gbit or even 2G+). Os is linux. Is it be faster with using 1 tcp socket (for send and recv) or with using 2 uni-directed tcp sockets?

The test for this task is very like NetPIPE network benchmark - measure latency and bandwidth for sizes from 2^1 up to 2^13 bytes, each size sent and received 3 times at least (in teal task the number of sends is greater. both processes will be sending and receiving, like ping-pong maybe).

The benefit of 2 uni-directed connections come from linux:

http://lxr.linux.no/linux+v2.6.18/net/ipv4/tcp_input.c#L3847

3847/*
3848 *      TCP receive function for the ESTABLISHED state. 
3849 *
3850 *      It is split into a fast path and a slow path. The fast path is 
3851 *      disabled when:
...
3859 *      - Data is sent in both directions. Fast path only supports pure senders
3860 *        or pure receivers (this means either the sequence number or the ack
3861 *        value must stay constant)
...
3863 *
3864 *      When these conditions are not satisfied it drops into a standard 
3865 *      receive procedure patterned after RFC793 to handle all cases.
3866 *      The first three cases are guaranteed by proper pred_flags setting,
3867 *      the rest is checked inline. Fast processing is turned on in 
3868 *      tcp_data_queue when everything is OK.

All other conditions for disabling fast path is false. And only not-unidirected socket stops kernel from fastpath in receive

Answer 1

I know this doesn't directly answer your question, but I would suggest taking a look at something like ZeroMQ. There is an introductory article about it on lwn: 0MQ: A new approach to messaging.

I haven't gotten to try it out yet, but I've read up on it a bit and it looks like it might be what you're looking for. Why reinvent the wheel?