mutliprocessing.Pool.add_sync() eating up memory

Question

I want to use multithreading to make my script faster... I'm still new to this. The Python doc assumes you already understand threading and what-not.

So...

I have code that looks like this

from itertools import izip
from multiprocessing import Pool

p = Pool()
for i, j in izip(hugeseta, hugesetb):
    p.apply_async(number_crunching, (i, j))

Which gives me great speed!

However, hugeseta and hugesetb are really huge. Pool keeps all of the i s and j s in memory after they've finished their job (basically, print output to stdout). Is there any to del i, and j after they complete?

Answer 1

The del statement deletes object references, so can free up memory when the garbage collector runs.

from itertools import izip
from multiprocessing import Pool

p = Pool()
for i, j in izip(hugeseta, hugesetb):
    p.apply_async(number_crunching, (i, j))

del i, j

Answer 2

Not really an answer but I used Pool.imap()instead:

for i in p.imap(do, izip(Fastitr(seqsa, filetype='fastq'), \
        Fastitr(seqsb, filetype='fastq'))):
    pass

Which works beautifully and garbage collects as expected however it feels funny having a for loop with nothing but pass actually do something useful.