Socket ping-pong performance

Question

I have written two simple programs (tried it in C++ and C#).

This is pseudo code:

-------- Client ---------------

for(int i = 0; i < 200.000; i++)  
{  
    socket_send("ping")  
    socket_receive(buff)  
}  

--------- Server -------------

while(1)  
{  
    socket_receive(buff)  
    socket_send("pong")  
}  

I tried it on Windows.

Execution time of client is about 45 seconds. Can somebody explain me why this takes so long? I understand that if there were real network connection between client and server the time of one 'ping-pong' would be: generate_ping + send_via_network + generate_pong + send_via_network but here everything is done in 'local' mode.

Is there any way to make this inter process ping-pong faster using network sockets (I'm not asking about shared memory for example :) )

Answer 1

Is your machine single-CPU? Then a full task switch is required after every send. That takes time.

Even if your machine is multi-CPU, the network stack must use locks/mutexes to guarantee that only a single process attempts to update certain internal data structures at any one time. Other processes attempting communications will need to wait, e.g. in a spinloop, for the mutex to be released. That takes time. (A good network stack implementation will cause minimal interference between communication requests on different processes that are independent from each other -- but clearly that's not the case here!)

Also the network code almost certainly resides in the OS kernel, which requires switching CPU protection levels upon entry and exit. That takes time.

(I assume you mean "200,000" as in "two hundred thousand", not "200.000" as in "two hundred, with 3 digits of redundant precision after the decimal place". I realise the meanings of , and . are swapped in some languages with respect to my understanding, just making sure.)

Finally, is latency (which is what you're really measuring here) really that important for you? I expect you will find bandwidth to be fine -- i.e. you could transfer much more data (certainly I would expect up to 1 virtual memory page, typically around 4Kb) per send without using much more time.

Answer 2

Here rough code of producer in C#:

using System;
using System.Net;
using System.Net.Sockets;


namespace Producer
{    
    class Program
    {
        static void Main(string[] args)
        {
            if (args.Length != 2)
            {
                Console.WriteLine("Usage:");
                Console.WriteLine("Producer.exe <server_ip> <port>");
                return;
            }
            string ipAddress = args[0];
            string port = args[1];

            Console.WriteLine("IpAddress: " + ipAddress);
            Console.WriteLine("Port     : " + port);

            Socket s = new Socket(AddressFamily.InterNetwork, 
                           SocketType.Stream, ProtocolType.Tcp);
            s.Connect(new IPEndPoint(IPAddress.Parse(ipAddress), 
                           int.Parse(port)));

            Console.WriteLine("Press key to start pinging");
            Console.ReadKey();

            byte[] buff = new byte[1];
            Console.WriteLine("Pinging started");
            DateTime start = DateTime.Now;

            for (int i = 0; i < 200000; i++)
            {
                s.Send(buff);
                s.Receive(buff, 1, SocketFlags.None);
            }
            DateTime end = DateTime.Now;
            Console.WriteLine("Pinging finished in: " 
                               + (end - start).TotalSeconds);

            Console.ReadKey();
        }
    }
}

And here is server in C#:

using System;
using System.Net;
using System.Net.Sockets;

namespace Server
{
    class Program
    {
        static void Main(string[] args)
        {
            if (args.Length < 1)
            {
                Console.WriteLine("Usage:");
                Console.WriteLine("Server.exe <port>");
                return;
            }

            string port = args[0];

            //IPHostEntry ipHostInfo = Dns.Resolve(Dns.GetHostName());
            //IPAddress ipAddress = ipHostInfo.AddressList[0];
            IPAddress ipAddress = IPAddress.Parse("127.0.0.1");
            IPEndPoint m_LocalEndPoint = new IPEndPoint(ipAddress, 
                                                        int.Parse(port));

            try
            {
                // Create a TCP/IP socket.
                Socket s = new Socket(AddressFamily.InterNetwork, 
                               SocketType.Stream, ProtocolType.Tcp);
                s.Bind(m_LocalEndPoint);
                s.Listen(10);

                Console.WriteLine("IpAddress: " + ipAddress.ToString());
                Console.WriteLine("Port     : " + port);

                byte[] buff = new byte[1];

                Socket s2 = s.Accept();
                Console.WriteLine("connected");

                while (true)
                {
                    s2.Receive(buff, 1, SocketFlags.None);
                    s2.Send(buff);
                }
            }
            catch( Exception )
            {
            }
        }
    }
}

And here is output from producer (run on Windows):

Producer.exe 127.0.0.1 667
IpAddress: 127.0.0.1
Port     : 667
Press key to start pinging
nPinging started
Pinging finished in: 46,617032

I've got very similar results for program written in C++ using winsock2 (code similar to Linux sockets below)

The same written in C and started on Linux:

Producer:

#include <stdio.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <time.h>

int main(int argc, char** argv)
{
  int sockfd, portno, n;
  struct sockaddr_in serv_addr;
  struct hostent *server;
  char buff[1];
  time_t start, end;
  int i;

  if (argc != 3)
  {
  printf("Usage:\n");
  printf("%s <server_ip> <port>\n", argv[0]);
  return;
  }

  char* ipAddress = argv[1];
  char* port = argv[2];

  printf("IpAddress: %s\n", ipAddress);
  printf("Port     : %s\n", port);

  portno = atoi(port);
  sockfd = socket(AF_INET, SOCK_STREAM, 0);
  if (sockfd < 0) 
  printf("error - socket open\n");

  memset(&serv_addr, 0, sizeof(serv_addr));
  serv_addr.sin_family = AF_INET;
  serv_addr.sin_addr.s_addr = inet_addr(ipAddress);
  serv_addr.sin_port = htons(portno);
  if(0 > connect(sockfd,&serv_addr,sizeof(serv_addr)))
printf("error - connect\n");

  start = time(0);
  printf("ping-pong started\n");
  for(i = 0; i < 200000; i++)
  {
write(sockfd,buff,1);
read(sockfd,buff,1);
  }
  end = time(0);
  printf("finished in %d secs\n", end - start);

  return 0;
}

And Server: #include #include #include #include #include

int main(int argc, char** argv)
{
  int sockfd, newsockfd, portno, clilen;
  char buffer[1];
  struct sockaddr_in serv_addr, cli_addr;
  int n;

  if (argc < 2)
  {
printf("Usage:\n");
printf("%s <port>\n", argv[0]);
return;
  }

  portno = atoi(argv[1]);
  sockfd = socket(AF_INET, SOCK_STREAM, 0);
  if (sockfd < 0) 
  printf("error - socket open\n");

  memset(&serv_addr, 0, sizeof(serv_addr));
  serv_addr.sin_family = AF_INET;
  serv_addr.sin_addr.s_addr = INADDR_ANY;
  serv_addr.sin_port = htons(portno);


  if (0 > bind(sockfd, (struct sockaddr *) &serv_addr, sizeof(serv_addr))) 
printf("error - bind\n");

  listen(sockfd,5);
  clilen = sizeof(cli_addr);
  printf("waiting for connection...\n");
  newsockfd = accept(sockfd, (struct sockaddr *) &cli_addr, &clilen);
  if (newsockfd < 0) 
printf("error - accept\n");

  printf("connected\n");

  while(1)
  {
read(newsockfd,buffer,1);
write(newsockfd,buffer,1);
  }  
  return 0;
}

Output from Producer (Linux):

$ ./a.out 127.0.0.1 666
IpAddress: 127.0.0.1
Port     : 666
ping-pong started
finished in 5 secs

My machine is Intel Core Duo

What is going on on Windows??

Answer 3

Could this be a nagle issue? You're sending very small packets, and then immediately waiting for a response. The TCP stack will hang on to the data for a bit until it is sure you're not going to send any more data. Setting the TCP_NODELAY option may make it faster.

What puzzles me about that hypothesis though is why no one else is seeing it and why the same effects aren't visible on linux (which I know virtually nothing about). I do know that the TCP stack on Windows is almost certainly not normally 9 times slower than the stack on linux - someone would have noticed.