How extract meaningful text from HTML

views:

123

answers:

+4 Q:

How extract meaningful text from HTML

I would like to parse a html page and extract the meaningful text from it. Anyone knows some good algorithms to do this?

I develop my applications on Rails, but I think ruby is a bit slow in this, so I think if exists some good library in c for this it would be appropriate.

Thanks!!

PD: Please do not recommend anything with java

UPDATE: I found this link text

Sadly, is in python

Lynx is able to do this. This is open source if you want to take a look at it.

mouviciel 2010-10-19 14:36:16

But spawning a separate program is not my idea of fast.

Amigable Clark Kant 2010-10-19 14:46:34

yes, you right. The website will crawl several pages and extract is text. The idea is to separate the text of the news the rest of the text. It must to be very fast.

Nisanio 2010-10-19 14:56:10

I don't suggest to use lynx as is. You can take whatever is of interest for you from the source code and compile it as a library.

mouviciel 2010-10-19 14:59:39

You should strip all angle-bracketed part from text and then collapse white-spaces. In theory the < and > should not be there in other cases. Pages contain < and > everywhere instead of them.

Collapsing whitespaces: Convert all TAB, newline, etc to spaces, then replace every sequence of spaces to a single space.

UPDATE: And you should start after finding the <body> tag.

Notinlist 2010-10-19 14:37:53

I would not recommend using regular expressions to parse HTML or any other format like it. (Except maybe trivial cases, but as a general rule, avoid.)

Amigable Clark Kant 2010-10-19 14:46:07

Regex + HTML: http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags/1732454#1732454

Nick T 2010-10-19 14:49:07

1st: @ Amigable Clark Kant: We are not talking about parsing, we are talking about stripping. A correct HTML can be stripped with regexp. If we have that in our specification then we can use it safely --- 2nd: You both misunderstood me. I did not recommend regexp for it. I expressed my idea about an algorithm and invoked the "regexp" phrase as a human language tool. I could write `<anything>`.

Notinlist 2010-10-19 15:06:52

Again: I did not recommended regexp!!! See editing history!

Notinlist 2010-10-21 13:33:23

+5 A:

Use Nokogiri, which is fast and written in C, for Ruby.

(Using regexp to parse recursive expressions like HTML is notoriously difficult and error prone and I would not go down that path. I only mention this in the answer as this issue seems to crop up again and again.)

With a real parser like for instance Nokogiri mentioned above, you also get the added benefit that the structure and logic of the HTML document is preserved, and sometimes you really need those clues.

Amigable Clark Kant 2010-10-19 14:41:48

+1 A:

Solutions integrating with Ruby

use Nokogiri as recommended by Amigable Clark kant
Use Hpricot

External Solutions

If your HTML is well-formed, you could use the Expat XML Parser for this.
For something more targeted toward HTML-only, the W3C actually released the code for the LibWWW, which contains a simple HTML parser (documentation).

haylem 2010-10-19 14:45:35

ansaurus

tags:

views:

answers:

How extract meaningful text from HTML

Solutions integrating with Ruby

External Solutions

related questions