The requirement is to cache a large number (100,000 to 1,000,000) of small objects and control the expiration of individual objects based on future events. There is no time component to the expiration.
The cache must be available to a number of application servers (SpringSource dm Server) and ideally should be distributable and error tolerant itself.
I have been looking at Terracotta as a possible solution. Is that a good path to go down? Is there a better one?
Some of the (more of less) new comers from the key-value world could be an alternative. If I understand correctly your requirements, memcached seams a natural fit, with a very small overhead. Client side consistent hashing will get you a reasonably good fault tolerancy.
Redis, Voldemort or Tokyo Tyrant can be relevant too, even if they more about storing than caching. Redis in particular has some very sexy features (list, set, atomic operations...)
We've been using Terracotta and we're extremely happy with it. Its Distributed Caches with the Cache Evictors were a great solution to our problems.
Have you considered Memcached? You've mentioned small objects specifically: memcached is great in that regard. It's distributed and fault tolerant, very fast, open source, actively used and maintained everywhere, and has two great Java client APIs: Danga and Spy.
The only requirement of yours that memcached doesn't meet is the lack of a time component to object expiration. I believe the maximum of amount of time an object can spend in cache is 30 days. YMMV.
Eric, I would recommend you to use NCache Enterprise , which lets you do what you are needing. It is a feature-rich distributed cache that gives you scalable performance even at peak load times. Yes you get options to expire, evict items from cache and you can define when you want to expire or evict them. It also has an option to sync your cache with database, use read-through and write-through caching for high data availability and even more. NCache Enterprise has 2 months evaluation period in which you can use all its features without limitations.
I think the best option for you is JBoss Infinispan (http://www.jboss.org/infinispan). The cache is stupid simple to setup, supports replication, eviction, persistence, among many other things. More importantly, the cache works in your application server. You can also access the cache using REST and there are abstractions for wiring Hibernate onto it.