How do I dump the TCP client's buffer in order to accept more data?

Question

I've got a simple TCP server and client. The client receives data:

received = sock.recv(1024)

It seems trivial, but I can't figure out how to recieve data larger than the buffer. I tried chunking my data and sending it multiple times from the server (worked for UDP), but it just told me that my pipe was broken.

Suggestions?

Answer 1

Keep in mind that:

• Your operating system has it's own idea of what it's TCP/IP socket buffer size is.
• TCP/IP packet maximum size (generally is 1500 bytes)
• pydoc for socket suggests that 4096 is a good buffer size

With that said, it'd really be helpful to see the code around that one line. There are a few things that could play into this, if you're using select or just polling, is the socket non-blocking, etc.

It also matters how you're sending the data, if your remote end disconnects. More details.

Answer 2

If you have no idea how much data is going to pour over the socket, and you simply want to read everything until the socket closes, then you need to put socket.recv() in a loop:

# Assumes a blocking socket.
while True:
    data = sock.recv(4096)
    if not data:
        break
    # Do something with `data` here.

Answer 3

Mike's answer is the one you're looking for, but that's not a situation you want to find yourself in. You should develop an over-the-wire protocol that uses a fixed-length field that describes how much data is going to be sent. It's a Type-Length-Value protocol, which you'll find again and again and again in network protocols. It future-proofs your protocol against unforeseen requirements and helps isolate network transmission problems from programmatic ones.

The sending side becomes something like:

socket.write(struct.pack("B", type)         #send a one-byte msg type 
socket.write(struct.pack("H", len(data))    #send a two-byte size field
socket.write(data)

And the receiving side something like:

type = socket.read(1)                                 # get the type of msg
dataToRead = struct.unpack("H", socket.read(2))[0]    # get the len of the msg
data = socket.read(dataToRead)                        # read the msg

if TYPE_FOO == type:
    handleFoo(data)

elif TYPE_BAR == type:
    handleBar(data)

else:
    raise UnknownTypeException(type)

You end up with an over-the-wire message format that looks like:

struct {
     unsigned char type;
     unsigned short length;
     void *data;
}