Zend_Search_Lucene crashes during indexing

Question

I wanted to create search engine for my webpage, but during indexing on server it crashes with errors :

Warning: opendir(/admin/lucene/) [function.opendir]: failed to open dir: Too many     open files in /admin/includes/Zend/Search/Lucene/Storage/Directory/Filesystem.php on line 159

Warning: readdir(): supplied argument is not a valid Directory resource in /admin/includes/Zend/Search/Lucene/Storage/Directory/Filesystem.php on line 160

Warning: closedir(): supplied argument is not a valid Directory resource in /admin/includes/Zend/Search/Lucene/Storage/Directory/Filesystem.php on line 167

Fatal error: Ignoring exception from Zend_Search_Lucene_Proxy::__destruct() while an exception is already active (Uncaught Zend_Search_Lucene_Exception in /admin/includes/Zend/Search/Lucene/Storage/File/Filesystem.php on line 66) in /admin/test.php on line 549

I am using newest version of ZF. Is there code solution for such error - I run script on localhost and it works great.

Thanks for any help.

Answer 1

PHP has hit the limit on the number of files it can have open at once it seems might be an option to change in php.ini, could be an OS (quota) limit or you might be able to tell the indexer to slow down and not have so many files open simultaneously.

Answer 2

This is most definitely a Linux/kernel imposed limitation. Use the following command as root on your machine:

cat /proc/sys/fs/file-nr

Return values are defined as:

1. Total allocated file descriptors
2. Total free allocated file descriptors
3. Maximum open file descriptors

I'm also going to take a guess and say you are on a shared hosting machine. If this is the case, I imagine that this sort of issue may come up frequently.

Finally, the following article provides a good amount of information on Linux and open file descriptors even if it is a little dated.

http://www.netadmintools.com/art295.html

Answer 3

It seems the problem is in the large number of segments in the index.

Could you check how much files does index folder contain?

There are two ways to solve this problem: a) Optimize index more often. b) Use another MaxBufferedDocs/MergeFactor parameters. See Zend_Search_Lucene documentation for details.

If it doesn't help, please register JIRA issue for the problem or email me directly ([email protected])