socket trouble in python

Question

I have a server that's written in C, and I want to write a client in python. The python client will send a string "send some_file" when it wants to send a file, followed by the file's contents, and the string "end some_file". Here is my client code :

file = sys.argv[1]
host = sys.argv[2]
port = int(sys.argv[3])
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((host,port))
send_str = "send %s" % file
end_str = "end %s" % file
sock.send(send_str)
sock.send("\n")
sock.send(open(file).read())
sock.send("\n")
sock.send(end_str)
sock.send("\n")

The problem is this :

• the server receives the "send some_file" string from a recv

• at the second recv, the file's content and the "end file" strings are sent together

In the server code, the buffer's size is 4096. I first noticed this bug when trying to send a file that's less than 4096k. How can I make sure that the server receives the strings independently?

Answer 1

With socket programming, even if you do 2 independent sends, it doesn't mean that the other side will receive them as 2 independent recvs.

One simple solution that works for both strings and binary data is to: First send the number of bytes in the message, then send the message.

Here is what you should do for each message whether it is a file or a string:

Sender side:

• Send 4 bytes that holds the number of bytes in the following send
• Send the actual data

Receiver side:

• From the receiver side do a loop that blocks on a read for 4 bytes
• Then do a block on a read for the number of characters specified in the preceding 4 bytes to get the data.

Along with the 4-byte length header I mentioned above, you could also add a constant size command type header (integer again) that describes what's in the following recv.

You could also consider using a protocol like HTTP which already does a lot of the work for you and has nice wrapper libraries.

Answer 2

TCP/IP data is buffered, more-or-less randomly.

It's just a "stream" of bytes. If you want, you can read it as though it's delimited by '\n' characters. However, it is not broken into meaningful chunks; nor can it be. It must be a continuous stream of bytes.

How are you reading it in C? Are you reading up to a '\n'? Or are you simply reading everything in the buffer?

If you're reading everything in the buffer, you should see the lines buffered more-or-less randomly.

If you read up to a '\n', however, you'll see each line one at a time.

If you want this to really work, you should read http://www.w3.org/Protocols/rfc959/. This shows how to transfer files simply and reliably: use two sockets. One for the commands, the other for the data.

Answer 3

In Socket Programming HOWTO, there is this quote and a code example:

A protocol like HTTP uses a socket for only one transfer. The client sends a request, the reads a reply. That's it. The socket is discarded. This means that a client can detect the end of the reply by receiving 0 bytes.

But if you plan to reuse your socket for further transfers, you need to realize that there is no "EOT" (End of Transfer) on a socket. I repeat: if a socket send or recv returns after handling 0 bytes, the connection has been broken. If the connection has not been broken, you may wait on a recv forever, because the socket will not tell you that there's nothing more to read (for now). Now if you think about that a bit, you'll come to realize a fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited (shrug), or indicate how long they are (much better), or end by shutting down the connection. The choice is entirely yours, (but some ways are righter than others).

Assuming you don't want to end the connection, the simplest solution is a fixed length message:

class mysocket:
    '''demonstration class only 
      - coded for clarity, not efficiency'''
    def __init__(self, sock=None):
        if sock is None:
            self.sock = socket.socket(
                socket.AF_INET, socket.SOCK_STREAM)
        else:
            self.sock = sock
    def connect(self,host, port):
        self.sock.connect((host, port))
    def mysend(self,msg):
        totalsent = 0
        while totalsent < MSGLEN:
            sent = self.sock.send(msg[totalsent:])
            if sent == 0:
                raise RuntimeError, \\
                    "socket connection broken"
            totalsent = totalsent + sent
    def myreceive(self):
        msg = ''
        while len(msg) < MSGLEN:
            chunk = self.sock.recv(MSGLEN-len(msg))
            if chunk == '':
                raise RuntimeError, \\
                    "socket connection broken"
            msg = msg + chunk
        return msg

Answer 4

Possibly using

sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

will help send each packet as you want it as this disables Nagle's algorithm, as most TCP stacks use this to join several packets of small-sized data together (and is on by default I believe)

Answer 5

There are two much simpler ways I can think of in which you can solve this. Both involve some changes in the behaviors of both the client and the server.

The first is to use padding. Let's say you're sending a file. What you would do is read the file, encode this into a simpler format like Base64, then send enough space characters to fill up the rest of the 4096-byte 'chunk'. What you would do is something like this:

from cStringIO import StringIO
import base64
import socket
import sys

CHUNK_SIZE = 4096 # bytes

# Extract the socket data from the file arguments
filename = sys.argv[1]
host = sys.argv[2]
port = int(sys.argv[3])
# Make the socket
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((host,port))
# Prepare the message to send
send_str = "send %s" % (filename,)
end_str = "end %s" % (filename,)
data = open(filename).read()
encoded_data = base64.b64encode(data)
encoded_fp = StringIO(encoded_data)
sock.send(send_str + '\n')
chunk = encoded_fp.read(CHUNK_SIZE)
while chunk:
    sock.send(chunk)
    if len(chunk) < CHUNK_SIZE:
        sock.send(' ' * (CHUNK_SIZE - len(chunk)))
    chunk = encoded_fp.read(CHUNK_SIZE)
sock.send('\n' + end_str + '\n')

This example seems a little more involved, but it will ensure that the server can keep reading data in 4096-byte chunks, and all it has to do is Base64-decode the data on the other end (a C library for which is available here. The Base64 decoder ignores the extra spaces, and the format can handle both binary and text files (what would happen, for example, if a file contained the "end filename" line? It would confuse the server).

The other approach is to prefix the sending of the file with the file's length. So for example, instead of sending send filename you might say send 4192 filename to specify that the length of the file is 4192 bytes. The client would have to build the send_str based on the length of the file (as read into the data variable in the code above), and would not need to use Base64 encoding as the server would not try to interpret any end filename syntax appearing in the body of the sent file. This is what happens in HTTP; the Content-length HTTP header is used to specify how long the sent data is. An example client might look like this:

import socket
import sys

# Extract the socket data from the file arguments
filename = sys.argv[1]
host = sys.argv[2]
port = int(sys.argv[3])
# Make the socket
sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
sock.connect((host,port))
# Prepare the message to send
data = open(filename).read()
send_str = "send %d %s" % (len(data), filename)
end_str = "end %s" % (filename,)
sock.send(send_str + '\n')
sock.send(data)
sock.send('\n' + end_str + '\n')

Either way, you're going to have to make changes to both the server and the client. In the end it would probably be easier to implement a rudimentary HTTP server (or to get one which has already been implemented) in C, as it seems that's what you're doing here. The encoding/padding solution is quick but creates a lot of redundantly-sent data (as Base64 typically causes a 33% increase in the quantity of data sent), the length prefix solution is also easy from the client side but may be more difficult on the server.