Develop a fast search functionality in a big forum?

Question

Hello!

I am building a forum from scratch in PHP. I have used the most of phpBB:s database structure.

But now I am thinking about the search functionality?, what is a good design to be able to search really fast in all posts. I guess there must be some better way than just %query_string% in mysql :)

Maybe explode all sentences into words, let the words be keys in a hash table, and the value is a comma separated list of all the post the word is in? Then there is little more trouble if you delete a post but I think that approach is better.

From start I guess I can use the simple solution, but I dont want to change the code when the forum grows bigger.

Thanks for any ideas or if you can point me to the right direction!

Answer 1

The best option for me today is sphinx search. It can be used with php, rails, perl and until now for me worked like a charm. You can check a php solution. Craiglist for example use it.

Answer 2

Zend Lucene is a powerful way to add search to a PHP site.

Here's an article about how to do exactly that: Roll Your Own Search Engine with Zend_Search_Lucene

Answer 3

Don't reinvent the wheel. Have a look at Lucene. There is also a port for php:

Zend Lucene

Lucene does the parsing and indexing for you and the queries are fast as lightning.

Answer 4

Most forum users will want more than just a string-search. They might not know the exact phrase they need and when they search for "forum search" they would be delighted to find a result for "How to search a forum", which contains the relevant terms but in a different order and separated by other words.

They may also need some fuzzy searching if they don't know the spelling of what they need. They might search for "sequal" and want "sql".

All of this points towards a more complex solution than your like-search.

The most important pointer for now is that whatever you implement, you should make sure it is easy to switch it out in favour of something better later. Make sure your search is hot-swappable as you know you will want to change it later.