ansaurus

Question

How might one go about implementing a forward index in PHP?

Answer 1

+3 A:

I don't think I'm totally clear on what you're trying to do, but you can get a simple result fairly easily:

Run the page through Tidy (a good introduction) to make sure it's going to have valid HTML.
Throw away everything before (and including) <body>.
Step through the document one character at a time.
1. If the character is a '<', don't do anything with the following characters until you see a '>' (skips HTML)
2. If the character is a "word character" (alphanumeric, hyphen, possibly more) append it to the "current word".
3. If the character is a "non-word character" (punctuation, space, possibly more), add the "current word" to the word list in the forward index, and clear the "current word".
Do the above until you hit </body>.

That's really about it, you might have to add in some exceptions for handling things like <script> tags (you don't want to consider javascript to be words that should be indexed), but that should give you a basic forward index.

Chad Birch 2009-04-27 22:34:41

Thanks Chad, that's really helpful :)

karim79 2009-04-27 22:38:57

Answer 2

+6 A:

Take a look at

http://simplehtmldom.sourceforge.net/

You do somthing like

$p = new Simple_dom_parser();
$p->load("www.page.com");
$p->find("body")->plaintext;

And that will give you all the text. Want to iterate over just the links

foreach ($p->find("a") as $link)
{
    echo $link->innerText;
}

It is very usefull and powerfull. Check it out.

Byron Whitlock 2009-04-27 23:46:01

ansaurus

tags:

views:

answers:

How might one go about implementing a forward index in PHP?

related questions