I am attempting to to use python to gain some performance on a task that can be highly parallelized using http://docs.python.org/library/multiprocessing.
When looking at their library they say to use chunk size for very long iterables. Now, my iterable is not long, one of the dicts that it contains is huge: ~100000 entries, with tuples as keys and numpy arrays for values.
How would I set the chunksize to handle this and how can I transfer this data quickly?
Thank you.
The only way to handle this single large item in multiple workers at once is by splitting it up. multiprocessing works by dividing up the work in units, but the smallest unit you can feed it is one object -- it can't know how to split up a single object in a way that's sensible. You have to do it yourself, instead. Instead of sending over the dicts to be worked on, split up the dicts to smaller work units and send those over instead. If you can't split the dict because all the data is interdependent, then you can't really split up the work either.