Ruby's Truncate unsanitizes MS Word code..

Question

Curious if anyone ever noticed this, but I have a WYSIWYG that users occassionally paste from word into. There is a word sanitizer, but not everyone's a genius.

If I parse that text somewhere else, it comes out right. But if I truncate it, then the msword code appears.

Does anyone know why truncate unsanitizes this || does anyone know how to sanitize and truncate at the same time?

UPDATE:

Here's an example of the msword being displayed after I truncate :

≪! [If Gte Mso 9]>≪Xml>  ≪Br /> ≪O:Office Document Settings>  ≪Br /> ≪O:Allow Png/>  ≪Br /> ≪/O:Office Document Settings>  ≪Br />≪/Xml>≪![Endif] >≪! [If Gte Mso 9]>≪Xml>  ≪Br /> ≪W:Word Document>  ≪Br /> ≪W:Zoom>0≪/W:Zoom>  ≪Br /> ≪W:Track Moves>False≪/W:Track Moves>  ≪Br /> ≪W:Track Formatting/>  ≪Br /> ≪W:Punctuation Kerning/>  ≪Br /> ≪W:Drawing Grid Horizontal Spacing>18 Pt≪/W:Drawing Grid Horizontal Spacing>  ≪Br /> ≪W:Drawing Grid Vertical Spacing>18 Pt≪/W:Drawing Grid Vertical Spacing>  ≪Br /> ≪W:Display Horizontal Drawing Grid Every>0≪/W:Display Horizontal Drawing Grid Every>  ≪Br /> ≪W:Display Vertical Drawing Grid Every>0≪/W:Display Vertical Drawing Grid Every>  ≪Br /> ≪W:Validate Against Schemas/>  ≪Br /> ≪W:Save If Xml Invalid>False≪/W:Save If Xml Invalid>  ≪Br /> ≪W:Ignore Mixed Content>False≪/W:Ignore Mixed Content>  ≪Br /> ≪W:Always Show Placeholder Text>False≪/W:Always Show Placeholder Text>  ≪Br /> ≪W:Compatibility>  ≪Br /> ≪W:Break Wrapped Tables/>  ≪Br /> ≪W:Dont Grow Autofit/>  ≪Br /> ≪W:Dont Autofit Constrained Tables/>  ≪Br /> ≪W:Dont Vert Align In Txbx/>  ≪Br /> ≪/W:Compatibility>  ≪Br /> ≪/W:Word Document>  ≪Br />≪/Xml>≪![Endif] >≪! [If Gte Mso 9]>≪Xml>  ≪Br /> ≪W:Latent Styles Def Locked State="False" Latent Style Count="276">  ≪Br /> ≪/W:Latent Styles>  ≪Br />≪/Xml>≪![Endif] >  ≪! {Cke Protected}%3 C!%2 D%2 D%7 Bcke Protected%7 D%253 C!%252 D%252 D%257 Bcke Protected%257 D%25253 C!%25252 D%25252 D%25257 Bcke Protected%25257 D%2525253 C!%2525252 D%2525252 D%2525257 Bcke Protected%2525257 D%252525253 C!%252525252 D%252525252 D%252525257 Bcke Protected%252525257 D%25252525253 C!%25252525252 D%25252525252 D%25252525257 Bcke Protected%25252525257 D%2525252525253 C!%2525252525252 D%2525252525252 D%2525252525250 A%25252525252520%2525252525252 F*%25252525252520 Font%25252525252520 Definitions%25252525252520*%2525252525252 F%2525252525250 A%25252525252540font Face%2525252525250 A%25252525252509%2525252525257 Bfont Family%2525252525253 A Times%2525252525253 B%2525252525250 A%25252525252509panose 1%2525252525253 A2%252525252525200%252525252525205%252525252525200%252525252525200%252525252525200%252525252525200%252525252525200%252525252525200%252525252525200%2525252525253 B%2525252525250 A%25252525252509mso Font Charset%2525252525253 A0%2525252525253 B%2525252525250 A%25252525252509mso Generic Font Family%2525252525253 Aauto%2525252525253 B%2525252525250 A%25252525252509mso Font Pitch%2525252525253 Avariable%2525252525253 B%2525252525250 A%25252525252509mso Font Signature%2525252525253 A3%252525252525200%252525252525200%252525252525200%252525252525201%252525252525200%2525252525253 B%2525252525257 D%2525252525250 A%25252525252540font Face%2525252525250 A%25252525252509%2525252525257 Bfont Family%2525252525253 A Verdana%2525252525253 B%2525252525250 A%25252525252509panose 1%2525252525253 A2%2525252525252011%252525252525206%252525252525204%25

The whole thing is about 600 characters long. This is the first 200 or so :

“Excellent” – The New York Times            

“4 Stars”  - The Star-Ledger                                                                       

“Best Romantic Restaurant” – Suburban Essex

“Best View” – OpenTable



In December 1986, the Knowles opened Highlawn after months of restoration to the former open-air “casino” which had, along with the now-prosperous park, been neglected for several years.

Here's a custom sanitizer I made with the help of Stackoverflow :

def sanitized_text(text)
  sanitized = text.gsub(/≪[^>]*>/, '')
end

The trouble with this sanitizer is that it returns empty white space after I truncate to 125 characters. I expanded it to 600 characters, and I get a single line that is another msword conditional statement.

Update: This is the code that produces the msword content.

 = truncate(organization.about_us, 125)

Note that when I just put this :

 = organization.about_us

It comes out fine, but of course not truncated.

I should also add this is Ruby 1.8.7 / rails 2.3.5

Answer 1

Truncating HTML is always a real hassle because you can end up splitting tags and entities. Without proper UTF-8 handling, you also run the risk of chopping a two byte character in half.

Another thing to watch out for is overly greedy regular expressions:

def sanitized_text(text)
  sanitized = text.gsub(/≪[^>]*?>/, '')
end

The *? will capture the minimum that matches, where * will capture the largest match.

For instance:

<A><B>

This can be grouped into "<", "A><B", and ">" if you end up with the wrong expression.

Edit: I've tried to reproduce this and had no luck.

With this example, using your text pasted in and sanitized, everything appears to be okay.

# app/controllers/example_controller.rb
class ExampleController < ApplicationController
  def index
    @text = '&Lt;! [If Gte Mso 9]>&Lt;Xml>  &Lt;Br /> &Lt;O:Office Document Settings>  &Lt;Br /> &Lt;O:Allow Png/>  &Lt;Br /> &Lt;/O:Office Document Settings>  &Lt;Br />&Lt;/Xml>&Lt;![Endif] >&Lt;! [If Gte Mso 9]>&Lt;Xml>  &Lt;Br /> &Lt;W:Word Document>  &Lt;Br /> &Lt;W:Zoom>0&Lt;/W:Zoom>  &Lt;Br /> &Lt;W:Track Moves>False&Lt;/W:Track Moves>  &Lt;Br /> &Lt;W:Track Formatting/>  &Lt;Br /> &Lt;W:Punctuation Kerning/>  &Lt;Br /> &Lt;W:Drawing Grid Horizontal Spacing>18 Pt&Lt;/W:Drawing Grid Horizontal Spacing>  &Lt;Br /> &Lt;W:Drawing Grid Vertical Spacing>18 Pt&Lt;/W:Drawing Grid Vertical Spacing>  &Lt;Br /> &Lt;W:Display Horizontal Drawing Grid Every>0&Lt;/W:Display Horizontal Drawing Grid Every>  &Lt;Br /> &Lt;W:Display Vertical Drawing Grid Every>0&Lt;/W:Display Vertical Drawing Grid Every>  &Lt;Br /> &Lt;W:Validate Against Schemas/>  &Lt;Br /> &Lt;W:Save If Xml Invalid>False&Lt;/W:Save If Xml Invalid>  &Lt;Br /> &Lt;W:Ignore Mixed Content>False&Lt;/W:Ignore Mixed Content>  &Lt;Br /> &Lt;W:Always Show Placeholder Text>False&Lt;/W:Always Show Placeholder Text>  &Lt;Br /> &Lt;W:Compatibility>  &Lt;Br /> &Lt;W:Break Wrapped Tables/>  &Lt;Br /> &Lt;W:Dont Grow Autofit/>  &Lt;Br /> &Lt;W:Dont Autofit Constrained Tables/>  &Lt;Br /> &Lt;W:Dont Vert Align In Txbx/>  &Lt;Br /> &Lt;/W:Compatibility>  &Lt;Br /> &Lt;/W:Word Document>  &Lt;Br />&Lt;/Xml>&Lt;![Endif] >&Lt;! [If Gte Mso 9]>&Lt;Xml>  &Lt;Br /> &Lt;W:Latent Styles Def Locked State="False" Latent Style Count="276">  &Lt;Br /> &Lt;/W:Latent Styles>  &Lt;Br />&Lt;/Xml>&Lt;![Endif] >  &Lt;! {Cke Protected}%3 C!%2 D%2 D%7 Bcke Protected%7 D%253 C!%252 D%252 D%257 Bcke Protected%257 D%25253 C!%25252 D%25252 D%25257 Bcke Protected%25257 D%2525253 C!%2525252 D%2525252 D%2525257 Bcke Protected%2525257 D%252525253 C!%252525252 D%252525252 D%252525257 Bcke Protected%252525257 D%25252525253 C!%25252525252 D%25252525252 D%25252525257 Bcke Protected%25252525257 D%2525252525253 C!%2525252525252 D%2525252525252 D%2525252525250 A%25252525252520%2525252525252 F*%25252525252520 Font%25252525252520 Definitions%25252525252520*%2525252525252 F%2525252525250 A%25252525252540font Face%2525252525250 A%25252525252509%2525252525257 Bfont Family%2525252525253 A Times%2525252525253 B%2525252525250 A%25252525252509panose 1%2525252525253 A2%252525252525200%252525252525205%252525252525200%252525252525200%252525252525200%252525252525200%252525252525200%252525252525200%252525252525200%2525252525253 B%2525252525250 A%25252525252509mso Font Charset%2525252525253 A0%2525252525253 B%2525252525250 A%25252525252509mso Generic Font Family%2525252525253 Aauto%2525252525253 B%2525252525250 A%25252525252509mso Font Pitch%2525252525253 Avariable%2525252525253 B%2525252525250 A%25252525252509mso Font Signature%2525252525253 A3%252525252525200%252525252525200%252525252525200%252525252525201%252525252525200%2525252525253 B%2525252525257 D%2525252525250 A%25252525252540font Face%2525252525250 A%25252525252509%2525252525257 Bfont Family%2525252525253 A Verdana%2525252525253 B%2525252525250 A%25252525252509panose 1%2525252525253 A2%2525252525252011%252525252525206%252525252525204%2'
  end
end

# app/helpers/example_helper.rb
module ExampleHelper
  def sanitized_text(text)
    text.gsub(/&Lt;[^>]*>/, '')
  end
end

The view itself is pretty much what you have:

<!-- app/views/example/index.html.erb -->
<body>
  <strong>Original</strong>
  <div>
    <%= sanitized_text(@text) %>
  </div>
  <strong>Truncated</strong>
  <div>
    <%= truncate(sanitized_text(@text), :length => 125) %>
  </div>
  <strong>Truncated With Deprecated Option</strong>
  <div>
    <%= truncate(sanitized_text(@text), 125) %>
  </div>
</body>

This was on OS X with Ruby 1.8.7p174, Rails 2.3.5 using WEBrick to test.