A multi-part/threaded downloader via python?

Question

I've seen a few threaded downloaders online, and even a few multi-part downloaders (HTTP).

I haven't seen them together as a class/function.

If any of you have a class/function lying around, that I can just drop into any of my applications where I need to grab multiple files, I'd be much obliged.

If there is there a library/framework (or a program's back-end) that does this, please direct me towards it?

Answer 1

Threadpool by Christopher Arndt may be what you're looking for. I've used this "easy to use object-oriented thread pool framework" for the exact purpose you describe and it works great. See the usage examples at the bottom on the linked page. And it really is easy to use: just define three functions (one of which is an optional exception handler in place of the default handler) and you are on your way.

from http://www.chrisarndt.de/projects/threadpool/:

• Object-oriented, reusable design
• Provides callback mechanism to process results as they are returned from the worker threads.
• WorkRequest objects wrap the tasks assigned to the worker threads and allow for easy passing of arbitrary data to the callbacks.
• The use of the Queue class solves most locking issues.
• All worker threads are daemonic, so they exit when the main program exits, no need for joining.
• Threads start running as soon as you create them. No need to start or stop them. You can increase or decrease the pool size at any time, superfluous threads will just exit when they finish their current task.
• You don't need to keep a reference to a thread after you have assigned the last task to it. You just tell it: "don't come back looking for work, when you're done!"
• Threads don't eat up cycles while waiting to be assigned a task, they just block when the task queue is empty (though they wake up every few seconds to check whether they are dismissed).

Also available at http://pypi.python.org/pypi/threadpool, easy_install, or as a subversion checkout (see project homepage).