I'm new to Python and seems that the multiprocessing and threads module are not very interesting and suffer from the same problems such as threads in Perl. Is there a technical reason why the interpreter can't use lightweight threads such as posix threads to make an efficient thread implementation that really runs on several cores?
+16
A:
It is using POSIX threads. The problem is the GIL.
Note that the GIL is not part of the Python spec --- it's part of the CPython reference implementation. Jython, for example, does not suffer from this problem.
That said, looked into Stackless ?
Alex Brasetvik
2009-12-16 12:25:57
Also, the GIL makes Cpython faster - Many tried but removing it seems to degrade performance.
nosklo
2009-12-16 14:35:54
nosklo: it may be more accurate to say that the GIL makes single-threaded CPython faster. modifications that removed it managed to make multi-threaded code of parallel algorithms run faster than the equivalent single-threaded implementations, but the speed of single-threaded code was considerably reduced
Eli Bendersky
2009-12-16 17:15:19
Certainly a pity given the importance of multicore chips and scalability.
piotr
2009-12-16 17:15:54
@piotr http://code.google.com/p/unladen-swallow/wiki/ProjectPlan#Global_Interpreter_Lock The GIL isn't the only thing hindering Python's scalability.
ephemient
2009-12-16 19:08:56
A:
Piotr,
You might want to take a look at stackless (http://www.stackless.com/) which is a modified version of python running lightweight tasklets in message passing (erlang style) fashion.
I'm not sure if you're looking for a multicore solution, but poking around in stackless might give you what you're looking for.
Ben
Ben Hughes
2009-12-16 18:50:53
Not by itself - you might consider twisted for this, although if you were looking for true multicore scalability (without distributing your application) Python might not be the best choice - you may want to consider Erlang.
Ben Hughes
2009-12-17 06:52:37
I've been reading the reasonings about process scalability. So they say if you really want to scale further than the local cores you will need several process anyway and thus the GIL is not a big deal. Thanks for the Erlang hint, seems that Erlang also does it by process.
piotr
2009-12-17 09:08:56