File encodings with ruby

Question

Hi, I'm having a bit problems with file encodings.

I'm receiving a url-encoded string like "sometext%C3%B3+more+%26+andmore", unescape it, process the data, and save it with windows-1252 encoding.

The conversions are these:

irb(main) >> value
=> "sometext%C3%B3+more+%26+andmore"
irb(main) >> CGI::unescape(value)
=> "sometext\303\263 more & andmore"
irb(main) >> #Some code and saved into a file using open(filename, "w:WINDOWS-1252")
irb(main) >> # result in the file:
=> sometextĂ³ more & andmore

And the result should be sometextó more & andmore

Answer 1

Encoding support has been added to Ruby 1.9, so the following code is from Ruby 1.9.1:

require 'cgi'
#=> true
s = "sometext%C3%B3+more+%26+andmore"
#=> "sometext%C3%B3+more+%26+andmore"
t = CGI::unescape s
#=> "sometext\xC3\xB3 more & andmore"
t.force_encoding 'utf-8' # telling Ruby that the string is UTF-8 encoded
#=> "sometextó more & andmore"
t.encode! 'windows-1252' # changing encoding to windows-1252
#=> "sometext? more & andmore"
# here you do whatever you want to do with windows-1252 encoded string

Here you have lots of informations on Ruby and encodings.

PS. Ruby 1.8.7 doesn't have built-in support for encodings, so you have to use some external library for conversion, for example iconv:

require 'iconv'
#=> true
require 'cgi'
#=> true
s = "sometext%C3%B3+more+%26+andmore"
#=> "sometext%C3%B3+more+%26+andmore"
t = CGI::unescape s
#=> "sometext\303\263 more & andmore"
Iconv.conv 'windows-1252', 'utf-8', t
#=> "sometext\363 more & andmore"
# \363 is ó in windows-1252 encoding