Python memory usage? loading large dictionaries in memory

Question

hey all, I have a file on disk that's only 168MB. It's just a comma separated list of word,id the word can be 1-5 words long. There's 6.5 million lines. I created a dictionary in python to load this up into memory so I can search incoming text against that list of words. When python loads it up into memory it shows 1.3 GB's of RAM space used. Any idea why that is?

so let's say my word file looks like this...

1,word1
2,word2
3,word3

then add 6.5 million to that I then loop through that file and create a dictionary (python 2.6.1)

  def load_term_cache():
      """will load the term cache from our cached file instead of hitting mysql. If it didn't 
      preload into memory it would be 20+ million queries per process"""
      global cached_terms
      dumpfile = os.path.join(os.getenv("MY_PATH"), 'datafiles', 'baseterms.txt')
      f = open(dumpfile)
      cache = csv.reader(f)
      for term_id, term in cache:
          cached_terms[term] = term_id
      f.close()

Just doing that blows up the memory. I view activity monitor and it pegs the memory to all available up to around 1.5GB of RAM On my laptop it just starts to swap. Any ideas how to most efficiently store key/value pairs in memory with python?

thanks

UPDATE: I tried to use the anydb module and after 4.4 million records it just dies the floating point number is the elapsed seconds since I tried to load it

56.95
3400018
60.12
3600019
63.27
3800020
66.43
4000021
69.59
4200022
72.75
4400023
83.42
4600024
168.61
4800025
338.57

you can see it was running great. 200,000 rows every few seconds inserted until I hit a wall and time doubled.

  import anydbm
  i=0
  mark=0
  starttime = time.time()
  dbfile = os.path.join(os.getenv("MY_PATH"), 'datafiles', 'baseterms')
  db = anydbm.open(dbfile, 'c')
  #load from existing baseterm file
  termfile = os.path.join(os.getenv("MY_PATH"), 'datafiles', 'baseterms.txt.LARGE')
  for line in open(termfile):
    i += 1
    pieces = line.split(',')
    db[str(pieces[1])] = str(pieces[0])
    if i > mark:
      print i
      print round(time.time() - starttime, 2)
      mark = i + 200000
  db.close()

Answer 1

convert your data into a dbm (import anydbm, or use berkerley db by import bsddb ...), and then use dbm API to access it.

the reason to explode is that python has extra meta information for any objects, and the dict needs to construct a hash table (which would require more memory). you just created so many objects (6.5M) so the metadata becomes too huge.

import bsddb
a = bsddb.btopen('a.bdb') # you can also try bsddb.hashopen
for x in xrange(10500) :
  a['word%d' %x] = '%d' %x
a.close()

This code takes only 1 second to run, so I think the speed is OK (since you said 10500 lines per second). btopen creates a db file with 499,712 bytes in length, and hashopen creates 319,488 bytes.

With xrange input as 6.5M and using btopen, I got 417,080KB in ouput file size and around 1 or 2 minute to complete insertion. So I think it's totally suitable for you.

Answer 2

Take a look (Python 2.6, 32-bit version)...:

>>> sys.getsizeof('word,1')
30
>>> sys.getsizeof(('word', '1'))
36
>>> sys.getsizeof(dict(word='1'))
140

The string (taking 6 bytes on disk, clearly) gets an overhead of 24 bytes (no matter how long it is, add 24 to its length to find how much memory it takes). When you split it into a tuple, that's a little bit more. But the dict is what really blows things up: even an empty dict takes 140 bytes -- pure overhead of maintaining a blazingly-fast hash-based lookup take. To be fast, a hash table must have low density -- and Python ensures a dict is always low density (by taking up a lot of extra memory for it).

The most memory-efficient way to store key / value pairs is as a list of tuples, but lookup of course will be very slow (even if you sort the list and use bisect for the lookup, it's still going to be extremely slower than a dict).

Consider using shelve instead -- that will use little memory (since the data reside on disk) and still offer pretty spiffy lookup performance (not as fast as an in-memory dict, of course, but for a large amount of data it will be much faster than lookup on a list of tuples, even a sorted one, can ever be!-).