Good high-level python ftp/http lib?

Question

I'm looking for a good, high-level python ftp client/server library. I'm working on a project that has "evolved" a small http/ftp library on top of ftplib/urllib/urllib2 from what was originally one function, and almost none of it was designed to be built upon. So now it's time to refactor kind of seriously, and I'd like to just switch to a library. The thing I'd most like to not deal with is robust-retry logic (like, keep retrying 15 times, or keep retrying until 12pm).

The problem that we've got right now is that we have about 10 separate grab() and put() functions. Aesthetically speaking, I'd rather have one of each with optional arguments along the lines of try_until=datetime(2009, 10, 7, 19) or retrys=15. We work with both binary and text data, so the functions would have to be reasonably smart about that. And we do way more grabbing than putting, so I can deal without the puts.

urlgrabber looks like exactly what I want, but there doesn't seem to have been any development for the last couple years and I'm not sure how compatible it is with 2.6. Anybody got much experience with this? Or opinions?

Answer 1

URLgrabber appears to be very mature, and since it's used by yum (and thus many Unix systems), I would expect it to be very stable. Python 2.x is largely backward compatible. You might encounter some warnings, but I would expect it to work suitably under Python 2.6.

Answer 2

Depending on the sort of application you are writing, you might want to consider twisted python, as it has http server and client code built in. However, it is a rather large departure from standard procedural python programming.

The big advantage of twisted for you is that it can handle your client requests in the background, handles retries and is very scalable.

Update

For a quick script that interacts with servers, see this serverfault answer: http://serverfault.com/questions/66336/script-automation-login-enter-password-run-commands-save-output-locally

It recomends the tool expect

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial. Expect is also useful for testing these same applications. And by adding Tk, you can also wrap interactive applications in X11 GUIs.

Expect can make easy all sorts of tasks that are prohibitively difficult with anything else. You will find that Expect is an absolutely invaluable tool - using it, you will be able to automate tasks that you've never even thought of before - and you'll be able to do this automation quickly and easily.

Sounds good to me!