Ignore Iconv::IllegalSequence while using Ruby WWW::Mechanize

Question

Hello, I've encountered the Iconv::IllegalSequence error on some web pages when using mechanize lib. Is there a way to make mechanize just omit ill encoded characters and return the "cut" page? I'm aware of the related thread, but I'd rather discard some characters on the page, then re-implement encoding guessing. TIA

Answer 1

The solution is to change line 40 in util.rb from

Iconv.iconv(code, "UTF-8", s).join("")

to

Iconv.iconv("#{code}//IGNORE", "UTF-8", s).join("")

or

Iconv.conv("#{code}//IGNORE", "UTF-8", s)

Answer 2

Thank you very much!