Hi,
Fastest way to uniqify a list in Python without preserving order? I saw many complicated solutions on Internet - could they be faster then simply:
list(set([a,b,c,a]))
?
Hi,
Fastest way to uniqify a list in Python without preserving order? I saw many complicated solutions on Internet - could they be faster then simply:
list(set([a,b,c,a]))
?
Going to a set only works for lists such that all their items are hashable -- so e.g. in your example if c = []
, the code you give will raise an exception. For non-hashable, but comparable items, sorting the list, then using itertools.groupby
to extract the unique items from it, is the best available solution (O(N log N)). If items are neither all hashable, nor all comparable, your only "last ditch" solution is O(N squared)
.
You can code a function to "uniquify" any list that uses the best available approach by trying each approach in order, with a try
/except
around the first and second (and a return
of the result either at the end of the try
clause, or, elegantly, in an else
clause of the try
statement;-).
Tim Peters wrote a classic general cookbook recipe for this problem back in 2001 (before sets were introduced). Comments by Alex Martelli, Raymond Hettinger et alia are informative and include updating to use sets etc.