Parallel Processing in python

Question

Whats a simple code that does parallel processing in python 2.7? All the examples Ive found online are convoluted and include unnecessary codes.

how would i do a simple brute force integer factoring program where I can factor 1 integer on each core (4)? my real program probably only needs 2 cores, and need to share information.

I know that parallel-python and other libraries exist, but i want to keep the number of libraries used to a minimum, thus I want to use the thread and/or multiprocessing libraries, since they come with python

Answer 1

mincemeat is the simplest map/reduce implementation that I've found. Also, it's very light on dependencies - it's a single file and does everything with standard library.

Answer 2

A good simple way to start with parallel processing in python is just the pool mapping in mutiprocessing -- its like the usual python maps but individual function calls are spread out over the different number of threads.

Factoring is a nice example of this - you can brute-force check all the divisions spreading out over all available threads:

from multiprocessing import Pool
import numpy

numToFactor = 976

def isFactor(x):
    result = None
    div = (numToFactor / x)
    if div*x == numToFactor:
        result = (x,div)
    return result

if __name__ == '__main__':
    pool = Pool(processes=4)
    possibleFactors = range(1,int(numpy.floor(numpy.sqrt(numToFactor)))+1)
    print 'Checking ', possibleFactors
    result = pool.map(isFactor, possibleFactors)
    cleaned = [x for x in result if not x is None]
    print 'Factors are', cleaned

This gives me

Checking  [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
Factors are [(1, 976), (2, 488), (4, 244), (8, 122), (16, 61)]