Download and write .tar.gz files without corruption.

Question

I've tried numerous ways of downloading files, specifically .zip and .tar.gz, with Ruby and write them to the disk.

I've found that the file appears to be the same as the reference (in size), but the archives refuse to extract. What I'm attempting now is:

Thanks!

def download_request(url, filePath:path, progressIndicator:progressBar) 

file = File.open(path, "w+")
    begin
      Net::HTTP.get_response URI.parse(url) do |response|

        if response['Location']!=nil
          puts 'Direct to: ' + response['Location']
          return download_request(response['Location'], filePath:path, progressIndicator:progressBar)
        end

        # some stuff

        response.read_body do |segment|
          file.write(segment) 
          # some progress stuff.
        end
      end
    ensure
      file.close
    end
end

download_request("http://github.com/jashkenas/coffee-script/tarball/master", filePath:"tarball.tar.gz", progressIndicator:nil)

Answer 1

I'd recommend using open-uri in ruby's stdlib.

require 'open-uri'

open(out_file, 'w') do |out|
  out.write(open(url).read)
end

http://ruby-doc.org/stdlib/libdoc/open-uri/rdoc/classes/OpenURI/OpenRead.html#M000832

Make sure you look at the :progress_proc option to open as it looks like you want a progress hook.

Answer 2

The last time I got currupted files with Ruby was when I forgot to call file.binmode right after File.open. Took me hours to find out what was wrong. Does it help with your issue?

Answer 3

I've successfully downloaded and extracted GZip files with this code:

require 'open-uri'
require 'zlib'

open('tarball.tar', 'w') do |local_file|
  open('http://github.com/jashkenas/coffee-script/tarball/master/tarball.tar.gz') do |remote_file|
    local_file.write(Zlib::GzipReader.new(remote_file).read)
  end
end