ASP.NET Lucene Performance Improvements question

I have coded up an ASP.NET website and running on win'08 (remotely hosted). The application queries 11 very large Lucene indexes (each ~100GB). I open IndexSearchers on Page_load() and keep them open for the duration of the user session.

My questions:

The queries take a ~5 seconds to complete - understandable these are very large indexes - but users want faster responses. I was curious to squeeze out better performance. ( I did look over the Apache Lucene website and try some of the ideas over there). Interested in if & how you tweaked it further, especially ones from asp.net perspective.
One ideas was to use Solr instead of querying Lucene directly. But that seems counter-intuitive, introducing another abstraction in between and might add to the latency. Is it worth the headache in porting to Solr? Can anyone share some metrics on what improvement you got following a switch to Solr if it has been worth it.
Are there some key things that could be done in Solr that could be replicated to speed up response times?

The queries to the indexes are context dependant. I hold the 11 searchers open for the duration of the user session. I am guessing this is inefficient, but this was a rush-job. I want to re-engineer; If I understand correctly you would recommend migrating to Solr (presumably multicore). Thanks for responding!

Mikos 2010-04-09 19:25:36

Can you somehow reduce the number of indexes? Do all of the searches have to be performed sequentially? With TPL you can create future tasks with continuations and parallelize that. Or you could try pooling your IndexSearchers to take advantage of the warm up.

Mauricio Scheffer 2010-04-09 21:02:00

Cannot reduce/collapse them - each index serves a different purpose. For example: one index is a list of Publications, one in a set of experts, one is a set of affiliations/institutes and so on....each needs to be query-able separately, especially w.r.t context.

Mikos 2010-04-10 07:02:30

Mikos 2010-04-10 07:10:24

If you migrate to Solr, you would represent each of your 11 indexes as a core. In addition to that, you can *shard* out (distribute) some or all of your indexes to other boxes. I think you will get better performance and also benefit from the additional flexibility, however I'd recommend running some tests to see if it's worth the migration for *your* particular task, i.e. if you get 4s instead of 5s you might decide it's not worth the migration.

Mauricio Scheffer 2010-04-11 15:59:13

Could you clarify on what "isolating to a search tier" means? Should I use a separate physical machine? Or something else?

Mikos 2010-04-09 20:51:10

@GalaticJello's suggestion is exactly what Solr is :)

Mauricio Scheffer 2010-04-09 20:57:33

"Are there some key things that could be done in Solr that could be replicated to speed up response times?"... and those key things are listed above, if you would like to replicate them.

GalacticJello 2010-04-12 15:03:18

ansaurus

tags:

views:

answers:

ASP.NET Lucene Performance Improvements question

related questions