Python: Sending a large dictionary to a server

Question

Hi Pythonistas,

I have an application that should communicate status information to a server. This information is effectively a large dictionary with string keys.

The server will run a web application based on Turbogears, so the server-side method called accepts an arbitrary number of keyword arguments.

In addition to the actual data, some data related to authentication (id, password..) should be transmitted. One approach would be to simply urlencode a large dictionary containing all this and send it in a request to the server.

urllib2.urlencode(dataPlusId)

But actually, the method doing the authentication and accepting the data set does not have to know much about the data. The data could be transmitted and accepted transparently and handed over to another method working with the data.

So my question is: What is the best way to transmit a large dictionary of data to a server in general? And, in this specific case, what is the best way to deal with authentication here?

Answer 1

I think the best way is to encode your data in an appropriate transfer format (you should not use pickle, as it's not save, but it can be binary) and transfer it as a multipart post request

What I do not know if you can make it work with repoze.who. If it does not support sign in and function call in one step, you'll perhaps have to verify the credentials yourself.

If you can wrap your data in xml you could also use XML-RPC.

Answer 2

Why don't you serialize the dictionary to a file, and upload the file? This way, the server can read the object back into a dictionary .

Answer 3

Have you tried using pickle on the data ?

Answer 4

Do a POST of your python data (use binary as suggested in other answers) and handle security using your webserver. Apache and Microsoft servers can both do authentication using a wide variety of methods (SSL client certs, Password, System accounts, etc...)

Serialising/Deserialising to text or XML is probably overkill if you're just going to turn it back to dictionary again).

Answer 5

I agree with all the answers about avoiding pickle, if safety is a concern (it might not be if the sender gets authenticated before the data's unpickled -- but, when security's at issue, two levels of defense may be better than one); JSON is often of help in such cases (or, XML, if nothing else will do...!-).

Authentication should ideally be left to the webserver, as SpliFF recommends, and SSL (i.e. HTTPS) is generally good for that. If that's unfeasible, but it's feasible to let client and server share a "secret", then sending the serialized string in encrypted form may be best.

Answer 6

I'd personally use SimpleJSON at both ends and just post the "file" (it would really just be a stream) over as multipart data.

But that's me. There are other options.

Answer 7

Thanks for the suggestions, I will consider them in turn.