How to avoid UDP socket transfers limits?

Question

Hi all, I write a little program that sends file from client to server trough udp socket connection.. the program works correctly but if the file I transfer is larger than 8192 kb the stream stops and the file I receive is corrupt.. How can I avoid this limitation?

server.py

host = ...
port = ...
filename = ...

buf = 2048
addr = (host, port)

UDPSock = socket(AF_INET, SOCK_DGRAM)
UDPSock.bind(addr)

f = open(filename, 'wb')

block,addr = UDPSock.recvfrom(buf)
while block:
    if(block == "!END"): # I put "!END" to interrupt the listener
        break
    f.write(block)
    block,addr = UDPSock.recvfrom(buf)

f.close()
UDPSock.close()

client.py

host = ...
port = ...
filename = ...

buf = 2048
addr = (host, port)

UDPSock = socket(AF_INET, SOCK_DGRAM)

f = open(filename, 'rb')

block = f.read(buf)
while block:
        UDPSock.sendto(block, addr)
        block = f.read(buf)

UDPSock.sendto("!END", addr)
f.close()
UDPSock.close()

Answer 1

You have to split the file into smaller chunks before sending t. 8192 is quite big and I think you could get in trouble when sending this over the Internet, I would stick to 512 bytes instead. Also remember that UDP is not a reliable protocol, i.e. some of your packets might not come at all. I suggest using TCP for transferring files, it solves all the trouble you will have to solve yourself when using UDP.

Answer 2

it would probably be a good idea to use tcp for large file transfers.

What you would need to do with udp is split the file up into chunks, hanle lost chunks, retrying and out of order pkts.

With tcp you dont.

Answer 3

The 8192 limit you hit corresponds to the size of socket send & receive buffers in Windows. I'm surprised it works at all for files above 1460 bytes, as the packets would have been truncated to that or smaller before they were put onto the wire. Were you testing within the same host?

When you've figured that out, you will then have to deal with packets ariving out of order,, more than once, or not at all. Are you sure you have to use UDP? It's not for the uninitiated.

Answer 4

All you are doing is very poorly re-implementing TCP, the only reason to use UDP should be for 1-to-many transfers or a constantly running enterprise service bus middleware.

Using UDP as an unreliable uni-directional transport should be considered a very small niche solution after exhausting the existing well defined transports.

Note for large data transfers over an internet, lower case 'I', are still only reliable and you should usually add application framing and payload verification.

I would recommend investigating existing file transfer technologies such as FSP (UDP), HTTP (TCP), SPDY (optimised HTTP), or for integration with other communications rolling your own above an existing message-orientated-middleware system such as JBoss or 0MQ.

Answer 5

You still haven't explained why you must use UDP - it simply isn't designed for high volume data transfers as it doesn't have any (simple) congestion management.

If you are sending VoIP then when not simply transmit at real-time, not "as fast as possible"?

FWIW, typical VoIP systems packetise the data into 20ms chunks or thereabouts. Hence if you're using an voice codec like GSM which requires 13 kbps, you only need to chunk into 260 bits (~32 bytes) per packet, and send them every 0.02 seconds.

Answer 6

If you have to use UDP, look at http://en.wikipedia.org/wiki/Trivial_File_Transfer_Protocol

It defines a reliable protocol. If your context allows TCP you should prefer to use that.