I have a function that is side effect free and I would like to run for every element in an array and return an array with all of the results. I'm wondering if python has something built into to generate all of the values.
Thank you.
+7
A:
Try the Pool.map function from multiprocessing:
http://docs.python.org/library/multiprocessing.html#using-a-pool-of-workers
It's not multithreaded per-se, but that's actually good since multithreading is severely crippled in Python by the GIL.
samtregar
2010-04-01 18:47:06
Very cool, that last line sold me. I saw this multiprocessing library before but I thought that it was too heavy weight for my needs. I think that I see the light now :) Thank you.
Sandro
2010-04-01 23:58:25
A:
You can try OpenMp, though I don't know if there is a native python interface. You would probably need to write the routine in C/C++ and use swig to call it. OpenMp is very use to use for this sort of loop parallelism and is used by issuing compiler directives.
frankc
2010-04-01 18:47:43
+1
A:
You can use the multiprocessing python package (http://docs.python.org/library/multiprocessing.html). The cloud python package, available from PiCloud (http://www.picloud.com), offers a multi-processing map() function as well, which can offload your map to the cloud.
BrainCore
2010-04-01 18:48:02
A:
This functionality is not built in. However, someone has already implemented it.
Thomas
2010-04-01 18:49:33
+2
A:
I would think there would be no reason to have such a function. All Python threads have to execute on the same CPU. Assuming your map function has no I/O component, you would not see any speedup in processing (and would probably see a slowdown due to context switching).
Other posters have mentioned multiprocessing - that is probably a better idea.
danben
2010-04-01 18:49:54
So is python multithreading really that bad? Do you have any sources on this info?
Sandro
2010-04-01 20:44:49
It is not a question of "bad" vs "good" - I have mentioned something specific and concrete, which is that the Python interpreter will not allow threads to execute simultaneously on multiple processors. The result, given that a processor can only be doing one thing at a time, is that CPU operations from different threads must be interleaved on a single processor. Executing all of the instructions from one thread followed by all of the instructions from another thread can be no slower than executing instructions in round-robin fashion (and will indeed be faster, as in the first case there
danben
2010-04-01 20:59:32
A:
Maybe try the Unladen Swallow Python 3 implementation? That might be a major project, and not guaranteed to be stable, but if you're inclined it could work. Then list or set comprehensions seem like the proper functional structure to use.
Adam Nelson
2010-04-01 22:12:42