ehcache persist to disk issues

Question

I want to do something with ehcache in Java that I think should be extremely simple, but I've spent enough time frustrating myself with the docs...

1. Write a value to a disk persistent cache. Shut down.

2. Start up again and read that value.

Here is my Java function:

private static void testCacheWrite() {

 // create the cache manager from our configuration
 URL url = TestBed.class.getClass().getResource("/resource/ehcache.xml");
 CacheManager manager = CacheManager.create(url);

 // check to see if our cache exits, if it doesn't create it
 Cache testCache = null;
 if (!manager.cacheExists("test")) {
  System.out.println("No cache found. Creating cache...");
  int maxElements = 50000;

  testCache = new Cache("test", maxElements,
    MemoryStoreEvictionPolicy.LFU, true, null, true, 60, 30,
    true, Cache.DEFAULT_EXPIRY_THREAD_INTERVAL_SECONDS, null);

  manager.addCache(testCache);

  // add an element to persist
  Element el = new Element("key", "value");
  testCache.put(el);
  testCache.flush();
  System.out.println("Cache to disk. Cache size on disk: " + testCache.getDiskStoreSize());
 } else {
  // cache exists so load it
  testCache = manager.getCache("test");

  Element el = testCache.get("key");
  if (null == el) {
   System.out.print("Value was null");
   return;
  }
  String value = (String) el.getObjectValue();
  System.out.println("Value is: " + value);
 }


 manager.shutdown();
}

And here is my cache configuration (ehcache.xml):

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="../config/ehcache.xsd">
    <diskStore path="C:/mycache"/><!-- java.io.tmpdir -->
 <defaultCache maxElementsInMemory="10000" eternal="true"
  timeToIdleSeconds="120" timeToLiveSeconds="120" overflowToDisk="true"
  maxElementsOnDisk="10000000" diskPersistent="true"
  diskExpiryThreadIntervalSeconds="120" memoryStoreEvictionPolicy="LRU" />
</ehcache>

Even though I see test.index and test.data files on disk after the first run, output from this function is always the following (it never seems to load the cache from disk):

No cache found. Creating cache...
Cache to disk. Cache size on disk: 2

I must be doing something dumb here, but I 'm not sure what!

Answer 1

I think you should remove the manager.cacheExists(..) test and simply create your cache using testCache = manager.getCache("test"); instead of using new Cache(..). Even if your cache is diskPersistent, it won't exist until you get it the first time. (At least that's what I think as I'm only using getCache(..) and it does exactly what you are looking for)

Note:

You could also add something like this to make sure the cache exists:

Cache cache = manager.getCache(name);
if (cache == null) {
    throw new NullPointerException(String.format("no cache with name %s defined, please configure it in %s", name, url));
}

Note 2:

If your configuration file is called ehcache.xml, you shouldn't use CacheManager.create(url). Instead use the CacheManager singleton: I think I've confused using CacheManager.create(url) with and using new CacheManager(url). Still, you should use the singleton for ehcache.xml and new CacheManager(url) for anything else.

// ehcache.xml - shared between different invocations
CacheManager defaultManager = CacheManager.getInstance();
// others - avoid calling twice with same argument
CacheManager manager = CacheManager.create(url);

Using CacheManager.create(..) is problematic as it might completely ignore the passed URL if any of the create(..) methods or getInstance() have been called before:

public static CacheManager create(URL configurationFileURL) throws CacheException {
    synchronized (CacheManager.class) {
        if (singleton == null) {
            if (LOG.isDebugEnabled()) {
                LOG.debug("Creating new CacheManager with config URL: " + configurationFileURL);
            }
            singleton = new CacheManager(configurationFileURL);

        }
        return singleton;
    }
}

That's why I wouldn't recommend using any of the CacheManager.create(..) methods. Use CacheManager.getInstance() or new CacheManager(url).

Answer 2

Okay, well what I did to fix this was configure my cache using the configuration file. Here is the updated config:

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:noNamespaceSchemaLocation="../config/ehcache.xsd">
 <diskStore path="C:/mycache" /><!-- java.io.tmpdir -->
 <defaultCache maxElementsInMemory="10000" eternal="true"
  timeToIdleSeconds="120" timeToLiveSeconds="120" overflowToDisk="true"
  maxElementsOnDisk="10000000" diskPersistent="true"
  diskExpiryThreadIntervalSeconds="120" memoryStoreEvictionPolicy="LRU" />
 <cache name="test" maxElementsInMemory="500" eternal="true"
  overflowToDisk="true" timeToIdleSeconds="300" timeToLiveSeconds="600"
  diskPersistent="true" diskExpiryThreadIntervalSeconds="1"
  memoryStoreEvictionPolicy="LFU" />
</ehcache>

So basically I didn't use the constructor to define the cache.

I suppose this will work, but I still wonder why programatically defined caches can't persist on disk (especially since they are still written to disk!).

Thanks for the comments guys.

Answer 3

I suppose this will work, but I still wonder why programatically defined caches can't persist on disk (especially since they are still written to disk!)

My understanding is that a programmatically created cache (i.e. not declared in ehcache.xml) can use a DiskStore that can itself be persistent but this doesn't mean that this cache will be loaded automatically by the CacheManager uppon restart. Actually, I don't think the previously mentioned files do contain the cache parameters.

But, if you "recreate" the cache programmatically with the same parameters, you'll find the previously cached entries back from the DiskStore.

Answer 4

After spending some quality time with the debugger, I believe I have an answer for the OP.

The issue (at least from what I have seen) centers around the non-clustered disk cache files and how they get read back in. In the file net.sf.ehcache.store.compound.factories.DiskPersistentStorageFactory.java, the method:

    public DiskPersistentStorageFactory(Ehcache cache, String diskPath) {
    super(getDataFile(diskPath, cache), cache.getCacheConfiguration().getDiskExpiryThreadIntervalSeconds(),
            cache.getCacheConfiguration().getDiskSpoolBufferSizeMB(), cache.getCacheEventNotificationService(), false);

    indexFile = new File(getDataFile().getParentFile(), getIndexFileName(cache));
    flushTask = new IndexWriteTask(indexFile, cache.getCacheConfiguration().isClearOnFlush());

    if (!getDataFile().exists() || (getDataFile().length() == 0)) {
        LOG.debug("Matching data file missing (or empty) for index file. Deleting index file " + indexFile);
        indexFile.delete();
    } else if (getDataFile().exists() && indexFile.exists()) {
        if (getDataFile().lastModified() > (indexFile.lastModified() + TimeUnit.SECONDS.toMillis(1))) {
            LOG.warn("The index for data file {} is out of date, probably due to an unclean shutdown. " 
                    + "Deleting index file {}", getDataFile(), indexFile);
            indexFile.delete();
        }
    }

    diskCapacity = cache.getCacheConfiguration().getMaxElementsOnDisk();
    memoryCapacity = cache.getCacheConfiguration().getMaxElementsInMemory();
    memoryPolicy = determineEvictionPolicy(cache.getCacheConfiguration());
}

checks the timestamps on the data files. The problem I am seeing is that no matter how I end up shutting down the cache/manager, the files are never get synchronized properly. My quick and dirty workaround was to adjust the time of the data file to be just past the timestamp on the index file:

        File index = new File( path, name + ".index" );
        File data  = new File( path, name + ".data"  );

        data.setLastModified( index.lastModified() + 1 );

Granted, this is not elegant, but it serves my needs, as our project uses clustered caches, and this allows me to debug standalone with a persistent cache...and without having to actually run Terracotta locally.

One caveat is that for non-clustered caches, I do have to flush() after every put() and remove() in order to keep the disk image fresh, especially when debugging due to the lack of shutdown support when you just "pull the plug".