HTML to Plain Text with Ruby?

Question

Is there anything out there to convert html to plain text (maybe a nokogiri script)? Something that would keep the line breaks, but that's about it.

If I write something on googledocs, like this, and run that command, it outputs (removing the css and javascript), this:

\n\n\n\n\nh1. Test h2. HELLO THEREI am some teexton the next line!!!OKAY!#*!)$!

So the formatting's all messed up. I'm sure someone has solved the details like these somewhere out there.

Answer 1

Is simply stripping tags and excess line breaks acceptable?

html.gsub(/<\/?[^>]*>/, '').gsub(/\n\n+/, "\n").gsub(/^\n|\n$/, '')

First strips tags, second takes duplicate line breaks down to one, third removes line breaks at the start and end of the string.

Answer 2

Actually, this is much simpler:

require 'rubygems'
require 'nokogiri'

puts Nokogiri::HTML(my_html).text

You still have line break issues, though, so you're going to have to figure out how you want to handle those yourself.

Answer 3

You could start with something like this:

require 'open-uri'
require 'rubygems'
require 'nokogiri'

uri = 'http://stackoverflow.com/questions/2505104/html-to-plain-text-with-ruby'
doc = Nokogiri::HTML(open(uri))
doc.css('script').each { |node| node.remove }
doc.css('link').each { |node| node.remove }
puts doc.css('body').text.squeeze(" ").squeeze("\n")

Answer 4

You want hpricot_scrub:

http://github.com/UnderpantsGnome/hpricot_scrub

You can specify which tags to strip / keep in a config hash.