views:

272

answers:

1

I'm unclear on why the sub-interpreter API exists and why it's used in modules such as the mod_wsgi apache module. Is it mainly used for creating a security sandbox for different applications running within the same process, or is it a way to allow concurrency with multiple threads? Maybe both? Are there other purposes?

+2  A: 

I imagine the purpose is to create separate python execution environments. mod_wsgi hosts a single python interpreter, and then hosts multiple applications within sub-interpreters.

Some key points from the documentation:

  • This is an (almost) totally separate environment for the execution of Python code. In particular, the new interpreter has separate, independent versions of all imported modules, including the fundamental modules __builtin__, __main__ and sys.
  • The table of loaded modules (sys.modules) and the module search path (sys.path) are also separate.
  • Because sub-interpreters (and the main interpreter) are part of the same process, the insulation between them isn’t perfect — for example, using low-level file operations like os.close() they can (accidentally or maliciously) affect each other’s open files.
  • Because of the way extensions are shared between (sub-)interpreters, some extensions may not work properly; this is especially likely when the extension makes use of (static) global variables, or when the extension manipulates its module’s dictionary after its initialization.
codeape
Does this mean that different interpreters can run concurrently in different threads? I'm still unclear on whether or not different interpreters in the same process share the same GIL.
James Whetstone
The GIL is a global object for the process, and is shared among the sub-interpreters. So no, they cannot run concurrently. http://objectmix.com/python/377035-multiple-independent-python-interpreters-c-c-program.html
codeape
Thanks for the link! I've been trying to figure out whether there's any way around the threading limitations of python and the GIL, and I'm not coming up with anything.
James Whetstone
The GIL only comes into play for execution of actual Python code. So, if you are using C extension modules which are able to do work without holding the GIL, you can still get some measure of concurrency. Some C extension modules deliberately partition their data so they can do this and thus get benefits of multi cpu/core systems.
Graham Dumpleton