Is there a way of keeping the index in RAM instead of keeping it on the hard disk?
We want to make searching faster.
views:
710answers:
3
+1
A:
A RAM disk could be a solution for this. A mini-HOWTO is available at http://www.vanemery.com/Linux/Ramdisk/ramdisk.html. Mount the RAM disk as your index directory, and you should be done.
pompo
2009-08-18 11:52:04
Note that for large indexes, "hardware" RAM disks are also available... basically device with a hard drive interface, but filled with DRAM instead of platters.
erickson
2009-08-18 16:03:51
Thank you for your answer. We considered that. But we tend not to use that option as we don't have that much experience in the operating system. We think that it would give us less control and less ability to observe/monitor the index.
elif
2009-08-19 08:01:56
A:
Check out the RAMDirectory documentation. Here's a basic usage example. This will only work if the index is small enough.
Yuval F
2009-08-18 12:11:34
And thank you for answering.What size would be small?The index is about 20 MB's and we don't expect it to grow more than 5 times. Is this small enough?
elif
2009-08-19 08:04:01
You're welcome. A 100 MB is indeed small enough. You should still consider using a disk index with proper JVM stack size settings. This will save you having to re-index whenever you restart the application, and may be as fast as the RAMDirectory. See also:http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Scaling-Lucene-and-Solr about search speed.
Yuval F
2009-08-19 09:03:21
Why does it reindex when I restart the application? I thought I could read the index on the hard disk into the RAM using RAMDirectory. Do I have to reindex everytime I want to read the index into the RAM?Thank you for the link.
elif
2009-08-20 10:24:09
You are right. You can read it from the hard disk:http://lucene.apache.org/java/2_4_1/api/org/apache/lucene/store/RAMDirectory.html#RAMDirectory%28java.io.File%29
Yuval F
2009-08-20 11:33:07
+5
A:
Is there a way of keeping the index in RAM instead of keeping it on the hard disk?
Using the RAMDirectory class
SampleUsage here
Also from the Lucene FAQs
Generally for faster indexing performance it's best to flush by RAM usage instead of document count and use as large a RAM buffer as you can.
EDIT: RE: RamDirectory, As the API says RamDirectory is A memory-resident Directory implementation., it keeps only those index in RAM as specified by directory RAMDirecory
RE:Caching In my knowledge, Lucene caches search results by means of filters pls look @ CachingWrapperFilter and QueryWrapperFilter
Narayan
2009-08-18 12:17:38
Thank you very much for your detailed answer. I have two further questions. First, I read that Lucene does caching, which would be keeping the index partially in RAM. But the RAMDirectory is different, right? Does it keep all of the index in the RAM? Second, I saw the setRAMBufferSizeMB, what I understood was that it was for speeding up the indexing rather than searching. Does it also speed up searching?
elif
2009-08-19 08:08:01