How to request for gzipped pages from web servers through ruby scripts?

Question

I have a ruby script that goes and saves web pages from various sites, how do i make sure that it checks if the server can send gzipped files and saves them if available... any help would be great!

Answer 1

You need to send the following header with your request:

Accept-Encoding: gzip,deflate

However, I am still reading how to code ruby and dont know how to do the header syntax in the net/http library (which I assume you are using to make the request)

Edit: Actually, according to the ruby doc it appears the this header is part of the default header sent if you dont specify other 'accept-encoding' headers.

Then again, like I said in my original answer, I am still just reading the subject so I could be wrong.

Answer 2

For grabbing web pages and doing stuff with them, ScrubyIt is terrific.

Answer 3

One can send custom headers as hashes ...

custom_request = Net::HTTP::Get.new(url.path, {"Accept-Encoding" => "gzip"})

you can then check the response by defining a response object as :

response = Net::HTTP.new(url.host, url.port).start do |http| http.request(custom_request) end

p [response['Content-Encoding']

Thanks to those who responded...