views:

4363

answers:

12

What is the most efficient Java Collections library?

A few years ago, I did a lot of Java and had the impression back then that trove is the best (most efficient) Java Collections implementation. But when I read the answers to the question "Most useful free Java libraries?" I noticed that trove is hardly mentioned. So which Java Collections library is best now?

UPDATE: To clarify, I mostly want to know what library to use when I have to store millions of entries in a hash table etc. (need a small runtime and memory footprint).

+20  A: 

From inspection, it looks like Trove is just a library of collections for primitive types - it's not like it's meant to be adding a lot of functionality over the normal collections in the JDK.

Personally (and I'm biased) I love the Google Java Collections. They make various tasks a lot easier, in a way which is at least reasonably efficient. Given that collection operations rarely form a bottleneck in my code (in my experience) this is "better" than a collections API which may be more efficient but doesn't make my code as readable.

Given that the overlap between Trove and the Google collections API is pretty much nil, perhaps you could clarify what you're actually looking for from a collections library.

Jon Skeet
should be mentioned that for most tasks google collections is too complex, and java collections are more than sufficient.
Andreas Petersson
@Andreas: Can't say I agree. Not that it's a "one or the other" scenario - I use the regular collections (with helpers like the Lists class) and then use Iterables etc when I need to. Use the complexity only when it helps you.
Jon Skeet
after reading my own comment several months after using G-C extensively - I disagree with my past opinion, and agree fully with yours. use the helper methods/classes extensively, they make much of the code more readable and safer.
Andreas Petersson
@Andreas: Thanks for coming back and saying so - I'm glad to hear that GJC is helping :)
Jon Skeet
+14  A: 

java.util

Sorry for the obvious answer, but for most uses, the default Java Collections are more than sufficient.

Yuval A
For basic uses, yes. But I think the framework misses some basic and advanced features (like immutable collections, filters, multimaps, etc) and that's where (for example) Google Collections comes in
Jorn
A: 

ConcurrentHashMap as well as the java.util.concurrent package should be mentioned, if you plan to use the HashMap in multiple threads. small memory footprint is assued, since this is part of standard java.

Andreas Petersson
+1  A: 

Depends on how we define "efficient".

Every data structure has its own Big-Oh behavior for reading, writing, iterating, memory footprint, etc. A linked list in one library is likely to be the same as any other. And a hash map will be faster for reading O(1) than a linked list O(n).

But when I read the answers to the question "Most useful free Java libraries?" I noticed that trove is hardly mentioned.

This doesn't sound like "most efficient". It sounds like "most popular" to me.

Just some feedback - I've never heard of it, and I don't know anyone who has used it. Collections built into the JDK, Google, or Apache Commons are well-known to me.

duffymo
+1  A: 

Trove offers a few advantages.

  • smaller memory footprint, it doesn't used Map.Entry objects
  • you can use hash strategies instead keys for maps, this saves memory and means you don't need to define a new key each time you want to cache an object on a new set of its attributes
  • it has primitive collection types
  • think it has some form of internal iterator

That said, a lot has been done to improve jdk collections since trove was written.

It's the hashing strategies that make it appealing to me though... Google for trove and read their overview.

+6  A: 

As other commentators have noticed, the definition of "efficient" casts a wide net. However no one has yet mentioned the Javolution library.

Some of the highlights:

  • Javolution classes are fast, very fast (e.g. Text insertion/deletion in O[Log(n)] instead of O[n] for standard StringBuffer/StringBuilder).
  • All Javolution classes are hard real-time compliant and have highly deterministic behavior (in the microsecond range). Furthermore (unlike the standard library), Javolution is RTSJ safe (no memory clash or memory leak when used with Java Real-Time extension).
  • Javolution's real-time collection classes (map, list, table and set) can be used in place of most standard collection classes and provide additional functionality.
  • The Javolution collections provide concurrency guarantees to make implementation of parallel algorithms easier.

The Javolution distribution includes a benchmark suite so you can see how they stack up against other libraries/the built-in collections.

sstock
+3  A: 

Some collection libs to consider:

I would first and foremost reach for the JDK collection library. It covers most common things you need to do and is obviously already available to you.

Google Collections is probably the best high-quality library outside the JDK. It's heavily used and well supported.

Apache Commons Collections is older and suffers a bit from the "too many cooks" problem but has a lot of useful stuff as well.

Trove has very specialized collections for cases like primitive keys/values. These days we find that on modern JDKs and with the Java 5+ collections and concurrent use cases, the JDK collections out-perform even the specialized Trove collections.

If you have really high concurrency use cases, you should definitely check out stuff like the NonBlockingHashMap in the high-scale lib, which is a lock-free implementation and can stomp on ConcurrentHashMap if you have the right use case for it.

Alex Miller
+2  A: 

If you want to store millions of records in a hash table, chances are that you will run into memory problems. This happened to me when I tried creating a map with 2.3 million String objects, for example. I went with BerkeleyDB, which is very mature and performs well. They have a Java API that wraps the Collections API, so you can easily create arbitrarily large maps with very little memory footprint. Access will be slower though (as it is stored on disk).

Follow-up question: is there a decent (and efficient), well maintained, library for immutable collections? Clojure has excellent support for this, and it would be nice to have something similar for Java.

fred-o
Google collections adds immutable Collections.
the.duckman
+15  A: 

The question is (now) about storing lots of data, which can be represented using primitive types like int, in a Map. Some of the answers here are very misleading in my opinion. Let's see why.

I modified the benchmark from trove to measure both runtime and memory consumption. I also added PCJ to this benchmark, which is another collections library for primitive types (I use that one extensively). The 'official' trove benchmark does not compare IntIntMaps to Java Collection's Map<Integer, Integer>, probably storing Integers and storing ints is not the same from a technical point of view. But a user might not care about this technical detail, he wants to store data representable with ints efficiently.

First the relevant part of the code:

new Operation() {

     private long usedMem() {
        System.gc();
        return Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();
     }

     // trove
     public void ours() {
        long mem = usedMem();
        TIntIntHashMap ours = new TIntIntHashMap(SET_SIZE);
        for ( int i = dataset.size(); i-- > 0; ) {
           ours.put(i, i);
        }
        mem = usedMem() - mem;
        System.err.println("trove " + mem + " bytes");
        ours.clear();
     }

     public void pcj() {
        long mem = usedMem();
        IntKeyIntMap map = new IntKeyIntOpenHashMap(SET_SIZE);
        for ( int i = dataset.size(); i-- > 0; ) {
           map.put(i, i);
        }
        mem = usedMem() - mem;
        System.err.println("pcj " + mem + " bytes");
        map.clear();
     }

     // java collections
     public void theirs() {
        long mem = usedMem();
        Map<Integer, Integer> map = new HashMap<Integer, Integer>(SET_SIZE);
        for ( int i = dataset.size(); i-- > 0; ) {
           map.put(i, i);
        }
        mem = usedMem() - mem;
        System.err.println("java " + mem + " bytes");
        map.clear();
     }

I assume the data comes as primitive ints, which seems sane. But this implies a runtime penalty for java util, because of the auto-boxing, which is not neccessary for the primitive collections frameworks.

The runtime results (without gc() calls, of course) on WinXP, jdk1.6.0_10:

                      100000 put operations      100000 contains operations 
java collections             1938 ms                        203 ms
trove                         234 ms                        125 ms
pcj                           516 ms                         94 ms

While this might already seem drastic, this is not the reason to use such a framework.

The reason is memory performance. The results for a Map containing 100000 int entries:

java collections        oscillates between 6644536 and 7168840 bytes
trove                                      1853296 bytes
pcj                                        1866112 bytes

Java Collections needs more than three times the memory compared to the primitive collection frameworks. I.e. you can keep three times as much data in memory, without resorting to disk IO which lowers runtime performance by magnitudes. And this matters. Read highscalability to find out why.

In my experience high memory consumption is the biggest performance issue with Java, which of course results in worse runtime performance as well. Primitive collection frameworks can really help here.

So: No, java.util is not the answer. And "adding functionality" to Java collections is not the point when asking about efficiency. Also the modern JDK collections do not "out-perform even the specialized Trove collections".

Disclaimer: The benchmark here is far from complete, nor is it perfect. It is meant to drive home the point, which I have experienced in many projects. Primitive collections are useful enough to tolerate fishy API - if you work with lots of data.

the.duckman
Actually, I think your answer is misleading. Storing ints vs Integers is very different, and most likely the main reason for the increased memory usage. I agree a raw type collection framework could be useful, but it doesn't make trove or pcj "better" than java.util.
Jorn
The question is about storing int data efficiently. Not about storing Integers. For this task trove/pcj are more efficient, as I tried to show. Using Integers imposes runtime and memory inefficiencies. Since java.util doesn't allow usage of primitives, it is not the best choice for this task.
the.duckman
+1  A: 

To store millions of String in a map, take a look at http://code.google.com/p/flatmap

Adrian
A: 

Re immutable collections, is not Collections.unmodifiableCollection() & friends somehow different than you need?

Jase
+1  A: 

I'm developer of happy-collections from happy-collections on source-forge

  1. Event based collections
  2. Unmodifiable
  3. SortedList
  4. Cache
Andreas Hollmann