Fastest way to obtain the largest X numbers from a very large unsorted list?

Question

Hello everyone,

I'm trying to obtain the top say, 100 scores from a list of scores being generated by my program. Unfortuatly the list is huge (on the order of millions to billions) so sorting is a time intensive portion of the program.

Whats the best way of doing the sorting to get the top 100 scores?

The only two methods i can think of so far is either first generating all the scores into a massive array and then sorting it and taking the top 100. Or second, generating X number of scores, sorting it and truncating the top 100 scores then continue generating more scores, adding them to the truncated list and then sorting it again.

Either way I do it, it still takes more time than i would like, any ideas on how to do it in an even more efficient way? (I've never taken programming courses before, maybe those of you with comp sci degrees know about efficient algorithms to do this, at least that's what I'm hoping).

Lastly, whats the sorting algorithm used by the standard sort() function in c++?

Thanks,

-Faken

Edit: Just for anyone who is curious...

I did a few time trials on the before and after and here are the results:

Old program (preforms sorting after each outer loop iteration):

top 100 scores: 147 seconds
top  10 scores: 147 seconds
top   1 scores: 146 seconds
Sorting disabled: 55 seconds

new program (implementing tracking of only top scores and using default sorting function):

top 100 scores: 350 seconds <-- hmm...worse than before
top  10 scores: 103 seconds 
top   1 scores:  69 seconds 
Sorting disabled: 51 seconds

new rewrite (optimizations in data stored, hand written sorting algorithm):

top 100 scores: 71 seconds <-- Very nice!
top  10 scores: 52 seconds
top   1 scores: 51 seconds
Sorting disabled: 50 seconds

Done on a core 2, 1.6 GHz...I can't wait till my core i7 860 arrives...

There's a lot of other even more aggressive optimizations for me to work out (mainly in the area of reducing the number of iterations i run), but as it stands right now, the speed is more than good enough, i might not even bother to work out those algorithm optimizations.

Thanks to eveyrone for their input!

Answer 1

You want the absolute largest X numbers, so I'm guessing you don't want some sort of heuristic. How unsorted is the list? If it's pretty random, your best bet really is just to do a quick sort on the whole list and grab the top X results.

If you can filter scores during the list generation, that's way way better. Only ever store X values, and every time you get a new value, compare it to those X values. If it's less than all of them, throw it out. If it's bigger than one of them, throw out the new smallest value.

If X is small enough you can even keep your list of X values sorted so that you are comparing your new number to a sorted list of values, you can make an O(1) check to see if the new value is smaller than all of the rest and thus throw it out. Otherwise, a quick binary search can find where the new value goes in the list and then you can throw away the first value of the array (assuming the first element is the smallest element).

Answer 2

1. take the first 100 scores, and sort them in an array.
2. take the next score, and insertion-sort it into the array (starting at the "small" end)
3. drop the 101st value
4. continue with the next value, at 2, until done

Over time, the list will resemble the 100 largest value more and more, so more often, you find that the insertion sort immediately aborts, finding that the new value is smaller than the smallest value of the candidates for the top 100.

Answer 3

Place the data into a balanced Tree structure (probably Red-Black tree) that does the sorting in place. Insertions should be O(lg n). Grabbing the highest x scores should be O(lg n) as well.

You can prune the tree every once in awhile if you find you need optimizations at some point.

Answer 4

If you only need to report the value of top 100 scores (and not any associated data), and if you know that the scores will all be in a finite range such as [0,100], then an easy way to do it is with "counting sort"...

Basically, create an array representing all possible values (e.g. an array of size 101 if scores can range from 0 to 100 inclusive), and initialize all the elements of the array with a value of 0. Then, iterate through the list of scores, incrementing the corresponding entry in the list of achieved scores. That is, compile the number of times each score in the range has been achieved. Then, working from the end of the array to the beginning of the array, you can pick out the top X score. Here is some pseudo-code:

    let type Score be an integer ranging from 0 to 100, inclusive.
    let scores be an array of Score objects
    let scorerange be an array of integers of size 101.

    for i in [0,100]
        set scorerange[i] = 0

    for each score in scores
        set scorerange[score] = scorerange[score] + 1

    let top be the number of top scores to report
    let idx be an integer initialized to the end of scorerange (i.e. 100)

    while (top > 0) and (idx>=0):
        if scorerange[idx] > 0:
              report "There are " scorerange[idx] " scores with value " idx
              top =  top - scorerange[idx]
        idx = idx - 1;

Answer 5

Declare an array where you can put the 100 best scores. Loop through the huge list and check for each item if it qualifies to be inserted in the top 100. Use a simple insert sort to add an item to the top list.

Something like this (C# code, but you get the idea):

Score[] toplist = new Score[100];
int size = 0;
foreach (Score score in hugeList) {
   int pos = size;
   while (pos > 0 && toplist[pos - 1] < score) {
      pos--;
      if (pos < 99) toplist[pos + 1] = toplist[pos];
   }
   if (size < 100) size++;
   if (pos < size) toplist[pos] = score;
}

I tested it on my computer (Code 2 Duo 2.54 MHz Win 7 x64) and I can process 100.000.000 items in 369 ms.

Answer 6

You can do this in O(n) time, without any sorting, using a heap:

#!/usr/bin/python

import heapq

def top_n(l, n):
    top_n = []

    smallest = None

    for elem in l:
        if len(top_n) < n:
            top_n.append(elem)
            if len(top_n) == n:
                heapq.heapify(top_n)
                smallest = heapq.nsmallest(1, top_n)[0]
        else:
            if elem > smallest:
                heapq.heapreplace(top_n, elem)
                smallest = heapq.nsmallest(1, top_n)[0]

    return sorted(top_n)


def random_ints(n):
    import random
    for i in range(0, n):
        yield random.randint(0, 10000)

print top_n(random_ints(1000000), 100)

Times on my machine (Core2 Q6600, Linux, Python 2.6, measured with bash time builtin):

• 100000 elements: .29 seconds
• 1000000 elements: 2.8 seconds
• 10000000 elements: 25.2 seconds

Edit/addition: In C++, you can use std::priority_queue in much the same way as Python's heapq module is used here. You'll want to use the std::greater ordering instead of the default std::less, so that the top() member function returns the smallest element instead of the largest one. C++'s priority queue doesn't have the equivalent of heapreplace, which replaces the top element with a new one, so instead you'll want to pop the top (smallest) element and then push the newly seen value. Other than that the algorithm translates quite cleanly from Python to C++.

Answer 7

You can do it in Haskell like this:

largest100 xs = take 100 $ sortBy (flip compare) xs

This looks like it sorts all the numbers into descending order (the "flip compare" bit reverses the arguments to the standard comparison function) and then returns the first 100 entries from the list. But Haskell is lazily evaluated, so the sortBy function does just enough sorting to find the first 100 numbers in the list, and then stops.

Purists will note that you could also write the function as

largest100 = take 100 . sortBy (flip compare)

This means just the same thing, but illustrates the Haskell style of composing a new function out of the building blocks of other functions rather than handing variables around the place.

Answer 8

I answered this question in response to an interview question in 2008. I implemented a templatized priority queue in C#.

using System;
using System.Collections.Generic;
using System.Text;

namespace CompanyTest
{
    // Based on pre-generics C# implementation at
    //  http://www.boyet.com/Articles/WritingapriorityqueueinC.html
    // and wikipedia article
    //  http://en.wikipedia.org/wiki/Binary_heap
    class PriorityQueue<T>
    {
     struct Pair
     {
      T val;
      int priority;
      public Pair(T v, int p)
      {
       this.val = v;
       this.priority = p;
      }
      public T Val { get { return this.val; } }
      public int Priority { get { return this.priority; } }
     }
     #region Private members
     private System.Collections.Generic.List<Pair> array = new System.Collections.Generic.List<Pair>();
     #endregion
     #region Constructor
     public PriorityQueue()
     {
     }
     #endregion
     #region Public methods
     public void Enqueue(T val, int priority)
     {
      Pair p = new Pair(val, priority);
      array.Add(p);
      bubbleUp(array.Count - 1);
     }
     public T Dequeue()
     {
      if (array.Count <= 0)
       throw new System.InvalidOperationException("Queue is empty");
      else
      {
       Pair result = array[0];
       array[0] = array[array.Count - 1];
       array.RemoveAt(array.Count - 1);
       if (array.Count > 0)
        trickleDown(0);
       return result.Val;
      }
     }
     #endregion
     #region Private methods
     private static int ParentOf(int index)
     {
      return (index - 1) / 2;
     }
     private static int LeftChildOf(int index)
     {
      return (index * 2) + 1;
     }
     private static bool ParentIsLowerPriority(Pair parent, Pair item)
     {
      return (parent.Priority < item.Priority);
     }
     // Move high priority items from bottom up the heap
     private void bubbleUp(int index)
     {
      Pair item = array[index];
      int parent = ParentOf(index);
      while ((index > 0) && ParentIsLowerPriority(array[parent], item))
      {
       // Parent is lower priority -- move it down
       array[index] = array[parent];
       index = parent;
       parent = ParentOf(index);
      }
      // Write the item once in its correct place
      array[index] = item;
     }
     // Push low priority items from the top of the down
     private void trickleDown(int index)
     {
      Pair item = array[index];
      int child = LeftChildOf(index);
      while (child < array.Count)
      {
       bool rightChildExists = ((child + 1) < array.Count);
       if (rightChildExists)
       {
        bool rightChildIsHigherPriority = (array[child].Priority < array[child + 1].Priority);
        if (rightChildIsHigherPriority)
         child++;
       }
       // array[child] points at higher priority sibling -- move it up
       array[index] = array[child];
       index = child;
       child = LeftChildOf(index);
      }
      // Put the former root in its correct place
      array[index] = item;
      bubbleUp(index);
     }
     #endregion
    }
}

Answer 9

Here's the 'natural' C++ way to do this:

std::vector<Score> v;
// fill in v
std::partial_sort(v.begin(), v.begin() + 100, v.end(), std::greater<Score>());
std::sort(v.begin(), v.begin() + 100);

This is linear in the number of scores.

The algorithm used by std::sort isn't specified by the standard, but libstdc++ (used by g++) uses an "adaptive introsort", which is essentially a median-of-3 quicksort down to a certain level, followed by an insertion sort.

Answer 10

Since speed is of the essence here, and 40.000 possible highscore values is totally maintainable by any of today's computers, I'd resort to bucket sort for simplicity. My guess is that it would outperform any of the algorithms proposed thus far. The downside is that you'd have to determine some upper limit for the highscore values.

So, let's assume your max highscore value is 40.000:

Make an array of 40.000 entries. Loop through your highscore values. Each time you encounter highscore x, increase your array[x] by one. After this, all you have to do is count the top entries in your array until you have reached 100 counted highscores.

Answer 11

Median of medians algorithm.