subprocess.Popen() has inconsistent behavior between Eclipse/PyCharm and terminal execution

Question

The problem I'm having is with Eclipse/PyCharm interpreting the results of subprocess's Popen() differently from a standard terminal. All are using python2.6.1 on OSX.

Here's a simple example script:

import subprocess

args = ["/usr/bin/which", "git"]
print "Will execute %s" % " ".join(args)
try:
  p = subprocess.Popen(["/usr/bin/which", "git"], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  # tuple of StdOut, StdErr is the responses, so ..
  ret = p.communicate()
  if ret[0] == '' and ret[1] <> '':
    msg = "cmd %s failed: %s" % (fullcmd, ret[1])
    if fail_on_error:
      raise NameError(msg)
except OSError, e:
  print >>sys.stderr, "Execution failed:", e

With a standard terminal, the line:

ret = p.communicate()

gives me:

(Pdb) print ret
('/usr/local/bin/git\n', '')

Eclipse and PyCharm give me an empty tuple:

ret = {tuple} ('','')

Changing the shell= value does not solve the problem either. On the terminal, setting shell=True, and passing the command in altogether (i.e., args=["/usr/bin/which git"]) gives me the same result: ret = ('/usr/local/bin/git\n', ''). And Eclipse/PyCharm both give me an empty tuple.

Any ideas on what I could be doing wrong?

Answer 1

Ok, found the problem, and it's an important thing to keep in mind when using an IDE in a Unix-type environment. IDE's operate under a different environment context than the terminal user (duh, right?!). I was not considering that the subprocess was using a different environment than the context that I have for my terminal (my terminal has bash_profile set to have more things in PATH).

This is easily verified by changing the script as follows:

import subprocess
args = ["/usr/bin/which", "git"]
print "Current path is %s" % os.path.expandvars("$PATH")
try:
  p = subprocess.Popen(args, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  # tuple of StdOut, StdErr is the responses, so ..
  out, err = p.communicate()
  if err:
    msg = "cmd %s failed: %s" % (fullcmd, err)
except OSError, e:
  print >>sys.stderr, "Execution failed:", e

Under the terminal, the path includes /usr/local/bin. Under the IDE it does not!

This is an important gotcha for me - always remember about environments!