Ruby open-uri, returns error when opening a png URL

Question

I am making a crawler parsing images on the Gantz manga at http://manga.bleachexile.com/gantz-chapter-1.html and on.

I had success until my crawler tried to open a image (on chapt 273):

bad URI(is not URI?): http://static.bleachexile.com/manga/gantz/273/Gantz[0273]_p001[Whatever-Illuminati].png

BUT this url is valid I guess, because I can open from Firefox.. Any thoughts?

Partial code:

img_link = nav.page.image_urls.find {|x| x.include?("manga/gantz")}
img_name = RAILS_ROOT+"/public/#{nome}/#{cap}/"+nome+((template).sub('::cap::', cap.to_s).sub('::pag::', i.to_s))
img = File.new( img_name, 'w' )
img.write( open(img_link) {|f| f.read} )
img.close

Answer 1

It is not a valid uri. Only certain characters are allowed for uri's. By the way firefox like all browsers try to do as much as possible for the user instead of complaining when it does not look standard compliant.

It is valid in the following form:

open("http://static.bleachexile.com/manga/gantz/273/Gantz%5B0273%5D_p001%5BWhatever-Illuminati%5D.png") # => #<File:/tmp/open-uri20100226-3342-clj08a-0>

You could try to escape it like this:

uri.gsub(/\/.*/) do |t|
  t.gsub(/[^.\/a-zA-Z0-9\-_ ]/) do |c|
    "%#{ c[0]<16 ? "0" : "" }#{ c[0].to_s(16).upcase }"
  end.gsub(" ", "+")
end

But be carefull, if the website uses correct escaped uri's and you escape them a second time. The uri's wont point to the same location anymore.