Python: How to transfer varrying length arrays over a network connection

Question

Hi,

I need to transfer an array of varying length in which each element is a tuple of two integers. As an example:

path = [(1,1),(1,2)]
path = [(1,1),(1,2),(2,2)]

I am trying to use pack and unpack, however, since the array is of varying length I don't know how to create a format such that both know the format. I was trying to turn it into a single string with delimiters, such as:

msg = 1&1~1&2~
sendMsg = pack("s",msg) or sendMsg = pack("s",str(msg))

on the receiving side:

path = unpack("s",msg)

but that just prints 1 in this case. I was also trying to send 4 integers as well, which send and receive fine, so long as I don't include the extra string representing the path.

sendMsg = pack("hhhh",p.direction[0],p.direction[1],p.id,p.health)

on the receive side:

x,y,id,health = unpack("hhhh",msg)

The first was for illustration as I was trying to send the format "hhhhs", but either way the path doesn't come through properly.

Thank-you for your help. I will also be looking at sending a 2D array of ints, but I can't seem to figure out how to send these more 'complex' structures across the network.

Thank-you for your help.

Answer 1

If you include message length as part of the message, then you will know how much data to read. So the entire string should be read across the network.

In any case, perhaps it would help if you posted some of the code you are using to send data across the network, or at least provided more of a description.

Answer 2

Pack and unpack are mandatory? If not, you could use JSON and YAML.

Don't use pickle because is not secure.

Answer 3

While you can use pack and unpack, I'd recommend using something like YAML or JSON to transfer your data.

• Pack and unpack can lead to difficult to debug errors and incompatibilities if you change your interface and have different versions trying to communicate with each other.
• Pickle can give security problems and the pickle format might change between Python versions.

JSON is included in the standard Python distribution since 2.6. For YAML there is PyYAML.

Answer 4

You want some sort of serialization protocol. twisted.spread provides one such (see the Banana specification or Perspective Broker documentation). JSON or protocol buffers would be more verbose examples.

See also Comparison of data serialization formats.

Answer 5

Take a look at xdrlib, it might help. It's part of the standard library, and:

The xdrlib module supports the External Data Representation Standard as described in RFC 1014, written by Sun Microsystems, Inc. June 1987. It supports most of the data types described in the RFC.