Most frequently repeated numbers in a huge list of numbers

Question

I have a file which has a many random integers(around a million) each seperated by a white space. I need to find the top 10 most frequently occurring numbers in that file. What is the most efficient way of doing this in java? I can think of 1. Create a hash map, key is the integer from the file and the value is the count. For every number in the file, check if that key already exists in the hash map, if yes, value++, else make a new entry in hash 2. Make a BST, each node is the integer from the file. For every integer from the file see if there is a node in the BST if yes, do value++, value is part of the node.

I feel hash map is better option if i can come up with good hashing function, Can some one pl suggest me what is the best of doing this ? Is there is anyother efficient algo that i can use?

Answer 1

Java handles hashing. You don't need to write a hash function. Just start pushing stuff in the hash map.

Also, if this is something that only needs to run once (or only occasionally), then don't both optimizing. It will be fast enough. Only bother if it's something that's going to run within an application.

Answer 2

Edit #2:

Okay, I screwed up my own first rule--never optimize prematurely. The worst case for this is probably using a stock HashMap with a wide range--so I just did that. It still runs in like a second, so forget everything else here and just do that.

And I'll make ANOTHER note to myself to ALWAYS test speed before worrying about tricky implementations.

(Below is older obsolete post that could still be valid if someone had MANY more points than a million)

A HashSet would work, but if your integers have a reasonable range (say, 1-1000), it would be more efficient to create an array of 1000 integers, and for each of your million integers, increment that element of the array. (Pretty much the same idea as a HashMap, but optimizing out a few of the unknowns that a Hash has to make allowances for should make it a few times faster).

You could also create a tree. Each node in the tree would contain (value, count) and the tree would be organized by value (lower values on the left, higher on the right). Traverse to your node, if it doesn't exist--insert it--if it does, then just increment the count.

The range and distribution of your values would determine which of these two (or a regular hash) would perform better. I think a regular hash wouldn't have many "winning" cases though (It would have to be a wide range and "grouped" data, and even then the tree might win.

Since this is pretty trivial--I recommend you implement more than one solution and test speeds against the actual data set.

Edit: RE the comment

TreeMap would work, but would still add a layer of indirection (and it's so amazingly easy and fun to implement yourself). If you use the stock implementation, you have to use Integers and convert constantly to and from int for every increase. There is the indirection of the pointer to the Integer, and the fact that you are storing at least 2x as many objects. This doesn't even count any overhead for the method calls since they should be inlined with any luck.

Normally this would be an optimization (evil), but when you start to get near hundreds of thousands of nodes, you occasionally have to ensure efficiency, so the built-in TreeMap is going to be inefficient for the same reasons the built-in HashSet will.

Answer 3

Why use a hashtable? Just use an array that is the same size as the range of your numbers. Then you don't waste time executing the hashing function. Then sort the values after you're done. O(N log N)

Answer 4

This is the source for java.lang.Integer.hashCode(), which is the hashing function that will be used if you store your entries as a HashMap<Integer, Integer>:

public int hashCode() {
return value;
}

So in other words, the (default) hash value of a java.lang.Integer is the integer itself.

What is more efficient than that?

Answer 5

1. Allocate an array / vector of the same size as the number of input items you have
2. Fill the array from your file with numbers, one number per element
3. Put the list in order
4. Iterate through the list and keep track of the the top 10 runs of numbers that you have encountered.
5. Output the top ten runs at the end.

As a refinement on step 4, you only need to step forward through the array in steps equilivent to your 10th longest run. Any run longer than that will overlap with your sampling. If the tenth longest run is 100 elements long, you only need to sample element 100, 200, 300 and at each point count the run of the integer you find there (both forwards and backwards). Any run longer than your 10th longest is sure to overlap with your sampling.

You should apply this optimisation after your 10th run length is very long compared to other runs in the array.

A map is overkill for this question unless you have very few unique numbers each with a large number of repeats.

NB: Similar to gshauger's answer but fleshed out

Answer 6

If you have to make it as efficient as possible, use an array of ints, with the position representing the value and the content representing the count. That way you avoid autoboxing and unboxing, the most likely killer of a standard Java collection.

If the range of numbers is too large then take a look at PJC and its IntKeyIntMap implementations. It will avoid the autoboxing as well. I don't know if it will be fast enough for you, though.

Answer 7

The correct way to do it is with a linked list. When you insert an element, you go down the linked list, if its there you increment the nodes count, otherwise create a new node with count of 1. After you inserted each element, you would have a sorted list of elements in O(n*log(n)).

For your methods, you are doing n inserts and then sorting in O(n*log(n)), so your coefficient on the complexity is higher.

Answer 8

If the range of numbers is small (e.g. 0-1000), use an array. Otherwise, use a HashMap<Integer, int[]>, where the values are all length 1 arrays. It should be much faster to increment a value in an array of primitives than create a new Integer each time you want to increment a value. You're still creating Integer objects for the keys, but that's hard to avoid. It's not feasible to create an array of 2^31-1 ints, after all.

If all of the input is normalized so you don't have values like 01 instead of 1, use Strings as keys in the map so you don't have to create Integer keys.

Answer 9

HashMap

A million integers is not really a lot, even for interpreted languages, but especially for a speedy language like Java. You'll probably barely even notice the execution time. I'd try this first and move to something more complicated if you deem this too slow.

It will probably take longer to do string splitting and parsing to convert to integers than even the simplest algorithm to find frequencies using a HashMap.

Answer 10

Use a HashMap to create your dataset (value-count pairs) in memory as you traverse the file. The HashMap should give you close to O(1) access to the elements while you create the dataset (technically, in the worst case HashMap is O(n)). Once you are done searching the file, use Collections.sort() on the value Collection returned by HashMap.values() to create a sorted list of value-count pairs. Using Collections.sort() is guaranteed O(nLogn). For example:

public static class Count implements Comparable<Count> {
 int value;
 int count;
 public Count(int value) {
  this.value = value;
  this.count = 1;
 }
 public void increment() {
  count++;
 }
 public int compareTo(Count other) {
  return other.count - count;
 }
}

public static void main(String args[]) throws Exception {
 Scanner input = new Scanner(new FileInputStream(new File("...")));
 HashMap<Integer, Count> dataset = new HashMap<Integer, Count>();
 while (input.hasNextInt()) {
  int tempInt = input.nextInt();
  Count tempCount = dataset.get(tempInt);
  if (tempCount != null) {
   tempCount.increment();
  } else {
   dataset.put(tempInt, new Count(tempInt));
  }
 }

 List<Count> counts = new ArrayList<Count>(dataset.values());
 Collections.sort(counts);

Answer 11

Actually, there is an O(n) algorithm for doing exactly what you want to do. Your use case is similar to an LFU cache where the element's access count determines whether it syays in the cache or is evicted from it.

http://dhruvbird.blogspot.com/2009/11/o1-approach-to-lfu-page-replacement.html