ansaurus

Question

Eliminating duplicates - shall I use a set?

Answer 1

+2 A:

Only if the order doesn't matter (since sets are unordered), and if the types are hashable (which strings are).

Ignacio Vazquez-Abrams 2010-10-20 11:14:08

Sorry, just edited the original question.

Helper Method 2010-10-20 11:16:28

The answer still holds.

Ignacio Vazquez-Abrams 2010-10-20 11:19:11

Answer 2

A:

you can use dictionary to store your path.

from collections import defaultdict
h=defaultdict(str)
uniq=[]
for line in open("file"):
    if "pattern" in line:
       # code to extract path here.
       extractedpath= ......
       h[extractedpath.strip()] = "" #using dictionary to store unique values
       if extractedpath not in uniq:
           uniq.append(extractedpath) #using a list to store unique values

ghostdog74 2010-10-20 11:26:17

What's the point of a dictionary here? A dict where the keys are irrelevant (set to some dummy Value and never used) is essentially a poor man's semantically-wrong, relatively inefficient set.

delnan 2010-10-20 11:31:09

the dictionary is just to store unique path values while iterating over the file. I could have use a list as well..

ghostdog74 2010-10-20 11:35:00

@ghostdog74: As @deinan was saying, it would be more efficient to store the paths in a `set` instead of a `dict` since you're not using the associated value dictionaries require to be associated with each key (i.e. the `""` dummy value you're using). Regardless of which data structure you use, each dictionary key or set element can appear in the container only once, so there's no need for maintaining a separate `uniq` list -- just extract all the dictionary keys or set elements into a list after processing all the lines in the file.

martineau 2010-10-20 12:16:33

Answer 3

A:

Only you should have full path names everywhere and if you are in Windows, names can be various cases as they are case insensitive. Also in Python you can also use / instead of \ (yes: be carefull of escaping the backslashes).

If you are dealing actually with URLs, most of time domain.com, domain.com/, www.domain.com and http://www.domain.com mean same thing and you should deside how to normalize.

Tony Veijalainen 2010-10-20 11:27:04

What does any of this have to do with the question?

Ignacio Vazquez-Abrams 2010-10-20 11:28:27

set requires exact key, one different case letter, extra space or / etc. and you get duplicate entries

Tony Veijalainen 2010-10-20 11:30:54

Answer 4

+4 A:

def sorted_paths(filename):
    with open(filename) as f:
       gen = (matches(line) for line in f)
       s = set(match.group(1) for match in gen if match)
    return sorted(s)

SilentGhost 2010-10-20 11:30:49

If the matching group is the whole line, I think the two generators could be merged into one.

mg 2010-10-20 12:32:45

Wow, now that's an elegant solution :-).

Helper Method 2010-10-20 12:43:18

Answer 5

+2 A:

This is a good way of doing it, both in terms of performance and in terms of conciseness.

fas 2010-10-20 11:44:24

Yes, using a set to remove duplicates is becoming a common Python idiom, as long as the items are hashable. Before they were introduced into the language, dictionaries with dummy values were often used instead.

martineau 2010-10-20 12:21:30

ansaurus

tags:

views:

answers:

Eliminating duplicates - shall I use a set?

related questions