Nested Lists in Dict : Accessing members of list within the list of a dictionary

Question

def index_dir(self, base_path):
    num_files_indexed = 0
    allfiles = os.listdir(base_path)
    self._documents = os.listdir(base_path)
    num_files_indexed = len(allfiles)
    docnumber = 0
    self._inverted_index = collections.defaultdict(list)
    docnumlist = []
    for file in allfiles: 
            self.documents = [base_path+file] #list of all text files
            f = open(base_path+file, 'r')
            lines = f.read()

            tokens = self.tokenize(lines)
            docnumber = docnumber + 1
            for term in tokens:  
            # check if the key/term already exists in the dictionary, 
            # if yes, just add a new key value/term into the dict
                if term not in sorted(self._inverted_index.keys()):
                    newlist=[]
                    tf=1
                    self._inverted_index[term] = []
                    #self._inverted_index[term][docnumber] +=1
                    newlist.append(docnumber)
                    newlist.append(tf)
                    self._inverted_index[term].append(newlist) #appending list to a list  
                else:
                        if docnumber not in self._inverted_index.get(term):
                            newlist=[]
                            tf=1
                            newlist.append(docnumber)
                            newlist.append(tf)
                            self._inverted_index[term].append(newlist) 
            f.close()
    print '\n \n'
    print 'Dictionary contents: \n'
    for term in sorted(self._inverted_index):
        print term, '->', self._inverted_index.get(term)
    return num_files_indexed
    return 0

What I get from this code:
dictionary in this format:
term <- [[docnumber, term freq][docnumber, term freq]]
for ex: if the word cat occurs in doc 1.txt for three times and in Doc 3.txt twice:
I get:
cat <- [[1,1],[1,1],[1,1],[3,1][3,1]]
so, instead of getting [1,1] three times, I want [1,3] added to the list
I don't know how to get rid of repetitive members of the list and increment the term freq.
What I should get:
cat <- [[1,3],[3,2]] i.e. thrice in Doc 1 and twice in doc 3.
I have tried ways to work it out, but I get access errors all the time.

Thanks in advance.

Answer 1

>>> from itertools import groupby
>>> from operator import itemgetter
>>> cat = [[1,1],[1,1],[1,1],[3,1],[3,1]]
>>> [(k,len(list(v))) for k, v in groupby(cat,itemgetter(0))]
[(1, 3), (3, 2)]

will fix your code. But that doesn't solve the problem of why the code is doing the wrong thing in the first place! The solution is to use the collections.Counter class, which will do the work for you if you just feed it a list of words.

>>> words = "Lorem ipsum dolor sit ames, lorem ipsum dolor sit ames.".split(" ")
>>> Counter(words)
Counter({'ipsum': 2, 'sit': 2, 'dolor': 2, 'lorem': 1, 'ames.': 1, 'ames,': 1, 'Lorem': 1})
>>> Counter(map(str.lower, words))
Counter({'ipsum': 2, 'sit': 2, 'dolor': 2, 'lorem': 2, 'ames.': 1, 'ames,': 1})

Answer 2

final counts:

{'cat':[[1,3], [3,2]]}

Words in current document:

{'cat':3}

I like that you have chosen to use defaultdict. It makes the following possible and is faster then looping through the keys.

from collections import defaultdict
    all_word_counts = defaultdict(list)
    all_word_counts['cat'].append([1, 3])

First count the word frequency in a given document

word_count = defaultdict(int) #reset each document
for term in self.tokenize(lines):
    word_count[term] += 1

Before moving on to the next document update the all_word_counts

for word, count in word_count.iteritems():
    all_word_counts[word].append([docnumber, count])