Eclipse using multiple Python interpreters with execnet

Question

Hi there, I'm using the execnet package to allow communication between Python scripts interpreted by different Python interpreters.

The following code (test_execnet.py):

import execnet
    for python_version in ('python', 'python3'):
        try:
            gw = execnet.makegateway("popen//python="+python_version)
            ch = gw.remote_exec('channel.send(1/3)')
            res = ch.receive()
            print(python_version, ': ', res, sep ="")
        except:
            print('problems with ', python_version)

Runs perfectly in the command-line Terminal, showing the following output:

$ python3 test_execnet.py 
python: 0
python3: 0.333333333333

However, if I try to run the same code from within the Eclipse IDE, I get the following error:

'import site' failed; use -v for traceback
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 4, in <module>
  File "<string>", line 2, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/site-packages/execnet/gateway_base.py", line 8, in <module>
    import sys, os, weakref
  File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/os.py", line 380, in <module>
    from _abcoll import MutableMapping  # Can't use collections (bootstrap)
  File "/Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/_abcoll.py", line 54
    class Hashable(metaclass=ABCMeta):
                            ^
SyntaxError: invalid syntax
problems with  python
problems with  python3

NOTE:

• Eclipse Version: 3.6.0
• PyDev Interpreter configured for the project: python3
• "Preferences/Interpreter - Python"'s Python Interpreters:
  • python (/usr/bin/python)
  • python3 (/Library/Frameworks/Python.Framework/Versions/3.1/Resources/Python.app/Contents/MacOS/Python

---

EDIT:

I write a code to show the os.environ like this:

for python_version in ('python', 'python3'):
    try:
        import os
        for item in os.environ:
            print(item, '= ', os.environ[item])
    except:
        print('problems with ', python_version)

I got the following outputs:

• eclipse_output.txt
• terminal_output.txt

A FileMerge comparison of the files can be found at eclipse_output.txt vs. terminal_output.pdf.

Any hints? Thanks

Answer 1

'import site' failed; use -v for traceback

I have seen that when python was unable to find its landmark. Which that indicates there is a PYTHONHOME problem.

Check out http://docs.python.org/using/cmdline.html#envvar-PYTHONHOME maybe eclipse is screwing your environment up.

Edit:

Looked at your env dumps, looks like eclipse is definitely messing with PYTHONPATH, which will cause your child python processes to not work correctly. Basically what you have going on here is eclipse starts a python v2 instance with a PYTHONPATH pointing to the python v2 directories. Then you spawn a python v3 process which tries to load its landmark from the python v2 directories...
You need to find a way to have eclipse not mess with the PYTHONPATH. I am not sure what eclipse is trying to do by doing that, but it is certainly no friend when you want to spawn new python processes.

Answer 2

Hi Eduardo,

seems like pydev does site-customizations and particularly modifies things for interactive/console usage (judging from a very quick skim of http://github.com/aptana/Pydev/blob/master/plugins/org.python.pydev/PySrc/pydev_sitecustomize/sitecustomize.py ). This is not useful or fitting for execnet-mediated processes.

You could try to "del os.environ['PYTHONPATH']" before you invoke execnet.makegateway, or, to be more careful, just delete the sitecustomize part of it.

hth, holger