Does anybody knows fate of Global Interpreter Lock in Python 3.1 against C++ multithreading integration
+15
A:
GIL is still there in CPython 3.1; the Unladed Swallow projects aims (among many other performance boosts) to eventually remove it, but it's still a way from its goals, and is working on 2.6 first with the intent of eventually porting to 3.x for whatever x will be current by the time the 2.y version is considered to be done. For now, multiprocessing (instead of threading) remains the way of choice for using multiple cores in CPython (IronPython and Jython are fine too, but they don't support Python 3 currently, nor do they make C++ integration all that easy either;-).
Alex Martelli
2009-08-03 15:25:03
Thanks, for an answer.IronPython hopefully has multithread solution since integrated with CLR.But my task is to plug Python into existing cross-platform C++ application. That is why neither IronPython nor multiprocessing looks good.
Dewfy
2009-08-03 16:56:59
Your C++ application will not be affected by the GIL as long as all entry points from Python into it use the proper macro to allow free threading - only Python's own execution will be serialized (with the GIL being anyway dropped during I/O and I know of no similar helpers for Jython.
Alex Martelli
2009-08-03 18:22:33
+1
A:
The GIL is a good thing.
Just make your C++ application release the GIL while it is doing its multithreaded work. Python code will continue to run in the other threads, unspoiled. Only acquire the GIL when you have to touch python objects.
nosklo
2009-08-03 20:39:39
+1
A:
The GIL will not affect your code which does not use python objects. In Numpy, we release the GIL for computational code (linear algebra calls, etc...), and the underlying code can use multithreading freely (in fact, those are generally 3rd party libraries which do not know anything about python)
David Cournapeau
2009-08-04 01:17:51
But exactly what I want - to run multiple plugged scripts simultanously. This idea stuck even when two simultanously executed chunk of Python doesn't use common resources.
Dewfy
2009-08-04 07:48:22
A:
I guess there will always be a GIL. The reason is performance. Making all the low level access thread safe - means putting a mutex around each hash operation etc. is heavy. Remember that a simple statement like
self.foo(self.bar, 3, val)
Might already have at least 3 (if val is a global) hashtable lookups at the moment and maybe even much more if the method cache is not hot (depending on the inheritance depth of the class)
It's expensive - that's why Java dropped the idea and introduced hashtables which do not use a monitor call to get rid of its "Java Is Slow" trademark.
Lothar
2009-09-21 21:36:43
Any information on how Jython and IronPython solve the same problem?
Pavel Minaev
2009-09-21 21:48:27
@Pavel, IronPython uses .Net approach - only explicitly "declared" methods are thread safe, since it is dynamic language (provided by .Net 3.5) there is no difference between .py and C# code.
Dewfy
2009-09-22 07:46:44
@Lothar You example bound to implmentation of GIL, that is why I'm strongly disagree with "Might already have at least 3..." . Alternative, for example, can be apartment model - you starts some instance of Python in apartment and mix code with C++ as you want. Synchronization is response of programmer. When 2 or more threads needs collaboration you provides these on demand.
Dewfy
2009-09-22 09:06:17
Don't know what an apartment model is i guess you simply mean separated memory space. Yes thats how TCL is doing it but it would be just a different implementation style of the multiprocessing model. For me threads always mean shared memory and therefore you have to share the interpreter instance and python runtime. And the runtime and interpreter have a lot of internal structures that need to be protected. Even if you don't care if you allow a python program to crash the interpreter you need a GIL or some synchronization.
Lothar
2009-09-22 13:19:07
A:
Significant changes will occur in the GIL for Python 3.2. Take a look at the What's New for Python 3.2, and the thread that initiated it in the mailing list.
While the changes don't signify the end of the GIL, they herald potentially enormous performance gains.
Matt Joiner
2010-01-26 04:51:43
@Matt Joiner I'm attentively gaze at "Unladen Swallow" (http://code.google.com/p/unladen-swallow/) project. It is only solution in term of my question.
Dewfy
2010-01-26 13:33:40
@Dewfy, I've taken a look at unladen-swallow, and they openly admit they weren't as successful as they hoped. their efforts may be merged into python 3.3 however, http://www.python.org/dev/peps/pep-3146/
Matt Joiner
2010-01-27 12:13:00
@Matt Joiner, let's cross th finger for python 3.3 to be succeeded with multi threading
Dewfy
2010-01-27 14:19:47