search engine (solr/sphinx) question

Question

i want to make my threads content searchable with full text search engines like solr.

but i wonder one thing. should i index just the thread.title, thread.body and post.body or should i index username, created date, nr of posts, views, country, region and city too that belongs to thread?

i mean when an user search for a thread he will get hits returned containing thread title, 2 lines of body, which user has posted it, creation date, tags, and so on.

should i index all this information too? but then it would be pretty much the whole database. or should i just index the 3 first columns i mentioned for full text search.

and another question. when an user post a new thread, then i have to immidiately tell solr to add that row? if im not, how would it be searchable?

Answer 1

I have used Apache Lucene but I haven't used Apache Solr yet. So I'm extrapolating some of this answer. But Lucene indexing is what powers Solr so I assume it's pretty much the same.

I would add everything that you might want as part of the searchable content or returned as a result of a search.

Note that Lucene allows you to add fields to a document as "not analyzed" -- which means those fields aren't part of the searchable content. But the extra fields are returned when you do a search and it finds that document. The alternative is to include only the primary key and then you have to use that to do an SQL lookup after you have found matching documents.

Lucene also supports queries against specific fields in the index. So you could include all the fields in the index, but if you want a given search tomatch against only a subset of the fields, you can do that.

Finally, yes you have to keep the Lucene/Solr index in sync with the data in your database. You can use the DataImportHandler to help load batches of data from the RDBMS to the Solr server. Or you can use Solr's REST-like HTTP interface to post individual documents if you need them indexed in real time as new threads are created.