How to use twisted for downloading a remote file?

Question

I'm relatively new to twisted and I'm planning on using it to create a file downloader. It would accept a file url and a number of parts to download the file.

What I have in mind is to split the file into how many parts the user specified and download each parts through deferred and when it is done, all parts gets assembled.

But do I need a protocol for each file to be downloaded and have each protocol dispatch a defer to download each file's chunks?

Is there a twisted component to read the remote file that has a seek? I really don't have any idea where to start.

Answer 1

If your mention of a URL implies that the protocol in use is HTTP (and I hope HTTP 1.1;-), then you could use twisted's relatively new HTTP 1.1 client (discussed at length here, and from the fact that the issue was marked as fixed 9 months ago I assume the client is finally in -- I have not checked that), using HTTP 1.1's range requests to get "slices" of the file.

If you're stuck with HTTP 1.0, or a not fully compliant server, you may be out of luck; if you really mean the "U" part of "URL", i.e., you need a Universal solution across all kinds of protocols, the problem of course becomes much, much harder.