In Python, how might one log in, answer a web form via HTTP POST (not url-encoded), and fetch a returned XML file?

Question

I am basically trying to export a configuration file, once a week. While the product in question allows you to manually log in via a web client, enter some information, and get an XML file back when you submit, there's no facility for automating this. I can get away with using Python 2.5 (have used for a while) or 2.6 (unfamiliar) to do this.

1) I think I need to have some way to authenticate against the product. Although I can view the cookie in FireFox, when I looked through the actual cookie.txt file, it was not present. Didn't show up after clearing my private data and re-authenticating. Odd. Should I be shooting for the Cookie module, or could this be some arcane method of authentication that only looks like cookies? How would I know?

2) I think I need the httplib module to do the HTTP POST, but I don't know how to do the multipart/form-data encoding. Have I overlooked a handy option, or is this something where I must roll my own?

3) I assume I can get the XML file back in the HTTPesponse from httplib.

I've fetched web things via Python before, but not with POST, multipart encoding, and authentication in the mix.

Answer 1

urllib2 should cover all of this.

Here's a Basic Authentication example.

Here's a Post with multipart/form-data.

Answer 2

Try mechanize module.

Answer 3

You should look at the MultipartPostHandler:

http://odin.himinbi.org/MultipartPostHandler.py

And if you need to support unicode file names , see a fix at: http://peerit.blogspot.com/2007/07/multipartposthandler-doesnt-work-for.html