Why does my solr slave index keep growing?

Question

I have a 5-core solr 1.4 master that is replicated to another 5-core solr using solr replication as described here. All writes are done against the master and replicated to the slave intermittently. This is done using the following sequence:

1. Commit on each master core
2. Replicate on each slave core
3. Optimize on each slave core
4. Commit on each slave core

The problem I am having is that the slave seems to be keeping around old index files and taking up ever more disk space. For example, after 3 replications, the master core data directory looks like this:

$ du -sh *
145M    index

But the data directory on the slave of the same core looks like this:

$ du -sh *
300M    index
144M    index.20100621042048
145M    index.20100629035801
4.0K    index.properties
4.0K    replication.properties

Here's the contents of index.properties:

#index properties
#Tue Jun 29 15:58:13 CDT 2010
index=index.20100629035801

And replication.properties:

#Replication details
#Tue Jun 29 15:58:13 CDT 2010
replicationFailedAtList=1277155032914
previousCycleTimeInSeconds=12
timesFailed=1
indexReplicatedAtList=1277845093709,1277155253911,1277155032914
indexReplicatedAt=1277845093709
replicationFailedAt=1277155032914
lastCycleBytesDownloaded=150616512
timesIndexReplicated=3

The solrconfig.xml for this slave contains the default deletion policy:

[...]
<mainIndex>
    <unlockOnStartup>false</unlockOnStartup>
    <reopenReaders>true</reopenReaders>
    <deletionPolicy class="solr.SolrDeletionPolicy">
        <str name="maxCommitsToKeep">1</str>
        <str name="maxOptimizedCommitsToKeep">0</str>
    </deletionPolicy>
</mainIndex>
[...]

What am I missing?

Answer 1

It is useless to commit and optimize on the slaves. Since all the write operations are done on the master, it is the only place where those operations should occur.

This may be the cause of the problem: since you do an additional commit and optimize on the slaves, it keeps more commit points on the slaves. But this is only a guess, it should be easier to understand what happens with your full solrconfig.xml on both the master and the slaves.

Answer 2

I determined that the extra index.* directories seem to be left behind when I replicate after completely reloading the master. What I mean by "completely reloading" is stopping the master, deleting everything under [core]/data/*, restarting (at which point solr creates a new index), indexing all of our docs, then replicating.

Based on some additional testing, I have found that it seems to be safe to remove the other index* directories (other than the one specified in [core]/data/index.properties). If I'm not comfortable with that workaround I may decide to empty the slave index (stop; delete data/*; start) before replicating the first time after completely reloading the master.