Deal with download interruptions in Tcl

Question

I'm downloading a list of files using tcl's http package and was wondering what the best way to handle interruptions are. Right now the outline to my download procedure looks like this:

proc downloadFiles {url_list folder_location} {
    foreach {i} $url_list {
        regsub {.*/(.*)$} $i {\1} $name
        set out [open $folder_location/$name w+] //not worried about errors here
        if {[catch {set token [http::geturl $i -channel $out]}]} {
            puts "Could not download $i"
        } else {
            http::cleanup $token
            puts "Downloaded $i"
        }
        close $out
    }
}

The line I'm having problems is the catch statement:

catch {set token [http::geturl $i -channel $out]}

Apparently despite me cutting off my internet and stopping a download half way the catch statement still returns a 0 for errors. Is there someway to catch this?

Answer 1

Detecting that the network has temporarily vanished is difficult (deliberately so; that's how TCP/IP works and HTTP just sits on top of that). What you could do is try getting the expected length of the data from the HTTP headers and comparing that to the length of data actually received (you'll want to force binary mode for this, but you're probably dealing with binary data anyway).

To get the expected data length and currently-downloaded length, you need a bit of magic with upvar (the internal alias name, state, is arbitrary):

upvar #0 $token state
puts "Content-Length was $state(totalsize), downloaded $state(currentsize)"

Note however that many pages don't supply a content length so the totalsize field is zero. The http package only knows in those cases that it's got the end when it gets to the end.

When restarting a download, you'll want to send a Range header. That's not an explicitly supported one so you'll need to give it by the -headers option to geturl.

http::geturl $url -headers [list Range bytes=$whereYouGotTo-]

Yes, the format is indeed that funky.