Getting Python under control on OS X - setting up environment and libraries

Question

After starting out with Python on Ubuntu Linux, I've now for a good while been doing most of my sustained work on the Mac, currently OS X 10.6. Unfortunately I've neglected to give proper attention to how Python is installed there and ended up with:

• Python 2.6.1 (Mac default version?) in /usr/bin (also, 2.5.4, which I'm not sure how it got there)
• Python 2.6.5 installed via MacPorts in /opt/local/bin/. This is my default
• I use pip to install libraries, which end up in some ungodly place (something like /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ - SRSLY?)
• Otherwise, mostly TextMate, and Git for version control. Django and AppEngine etc...

Now I'm preparing to set up 2.7 and 3.2, and am unhappy with the haphazard state of things. So what are your favourite approaches to organising the code and libraries, and how have you wrestled the Mac into submission?

I'd like to continue using pip, but would like to have more control or at least understanding of what libraries for which version are getting installed and made accessible from where: I've had problems with installing py.test via pip and only being able to load it from the obsolete 2.6.1 Python, not my current 2.6.5 one. MacPorts has python_select, but it's not overly helpful:

reason: chris$ python_select -l
Available versions:
current none python26 python26-apple

Most Python people I've asked don't use MacPorts, which I don't much like, but the stock Mac Python from python.org. I've also heard the recommendation to use virtualenv systematically, so am interested in links to a good practical intro.

Answer 1

virutalenv is a great tool and is very useful for managing multiple Python instances on most platforms. On OS X, though, things are more complicated because the use of framework builds makes it common to encounter multiple instances of even the same major version of Python. I suggest you first understand and be comfortable with how to manage framework installs on OS X before delving into virtualenv.

"/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/ - SRSLY?"

Yes, that's the default location within a Python OS X framework build. Most distributors of Python on OS X use a framework build, each generally using a different root, because it makes it simpler to support multiple versions and to keep all the site packages installed for a particular Python instance together in one place.

For OS X 10.6, Apple supplies a Python 2.6 (rooted at /System/Library/Frameworks/Python.framework and symlinked to /usr/bin/python2.6) as well as a Python 2.5 (for compatibility with OS X 10.5 - that's the /usr/bin/python2.5 you see). The python.org OS X installers install to /Library/Frameworks/Python.framework. MacPorts uses /opt/local/Library/Frameworks/Python.framework.

The frameworks include a bin directory (at ../Versions/x.x/bin in the framework); this is normally the default location where package scripts will be installed. It's also where symlinks to the interpreter binaries are placed.

Here's a look at a python.org-style framework with multiple versions of Python installed:

$ cd /Library/Frameworks/Python.framework/Versions/
$ ls -l
total 8
drwxrwxr-x  8 root  admin  340 Dec 23  2008 2.5/
drwxrwxr-x  8 root  admin  340 Aug 10 21:42 2.6/
drwxrwxr-x  8 root  admin  340 Jul  3 12:24 2.7/
drwxrwxr-x  7 root  admin  306 Mar 23 17:42 3.1/
drwxrwxr-x  7 root  admin  306 Aug 10 22:14 3.2/
lrwxr-xr-x  1 root  admin    3 Aug 10 21:47 Current@ -> 2.6
$ cd 2.6
$ ls -l
total 7776
lrwxr-xr-x  1 root  admin       17 Aug 10 21:47 Headers@ -> include/python2.6
drwxrwxr-x  3 root  admin      102 Aug 10 21:42 Mac/
-r-xrwxr-x  1 root  admin  3973552 Aug 10 21:41 Python*
drwxrwxr-x  4 root  admin      170 Aug 10 21:48 Resources/
drwxrwxr-x  2 root  admin      578 Aug 12 15:19 bin/
drwxrwxr-x  3 root  admin      102 Aug 10 21:42 include/
drwxrwxr-x  3 root  admin      646 Aug 10 21:43 lib/
drwxrwxr-x  3 root  admin      102 Aug 10 21:41 share/
$ ls -l bin
total 264
-rwxrwxr-x  1 root  admin    150 Aug 10 21:41 2to3*
-rwxr-xr-x  1 nad   admin    360 Aug 12 15:19 easy_install*
-rwxr-xr-x  1 nad   admin    368 Aug 12 15:19 easy_install-2.6*
lrwxr-xr-x  1 root  admin      7 Aug 10 21:47 idle@ -> idle2.6
-rwxrwxr-x  1 root  admin    138 Aug 10 21:41 idle2.6*
lrwxr-xr-x  1 root  admin      8 Aug 10 21:47 pydoc@ -> pydoc2.6
-rwxrwxr-x  1 root  admin    123 Aug 10 21:41 pydoc2.6*
lrwxr-xr-x  1 root  admin      9 Aug 10 21:47 python@ -> python2.6
lrwxr-xr-x  1 root  admin     16 Aug 10 21:47 python-config@ -> python2.6-config
-rwxrwxr-x  1 root  admin  29584 Aug 10 21:43 python2.6*
-rwxrwxr-x  1 root  admin   1463 Aug 10 21:42 python2.6-config*
lrwxr-xr-x  1 root  admin     10 Aug 10 21:47 pythonw@ -> pythonw2.6
-rwxrwxr-x  1 root  admin  29584 Aug 10 21:43 pythonw2.6*
lrwxr-xr-x  1 root  admin     11 Aug 10 21:47 smtpd.py@ -> smtpd2.6.py
-rwxrwxr-x  1 root  admin  18272 Aug 10 21:41 smtpd2.6.py*

Here's a MacPorts bin including a number of additional site packages:

$ cd /opt/local/Library/Frameworks/Python.framework/Versions/2.6
$ ls -l bin
total 952
-rwxr-xr-x  2 root  wheel    160 Jul 15 01:56 2to3*
-rwxr-xr-x  2 root  wheel   1541 Jun  8 19:59 apirst2html.py*
-rwxr-xr-x  2 root  wheel    363 Oct  5  2009 bdist_mpkg*
[...]
-rwxr-xr-x  2 root  wheel    211 Jun  2 12:45 django-admin.py*
-rwxr-xr-x  2 root  wheel    410 Jul 15 17:30 easy_install*
-rwxr-xr-x  2 root  wheel    418 Jul 15 17:30 easy_install-2.6*
[...]
-rwxr-xr-x  2 root  wheel    368 Jun 24 02:00 pip*
-rwxr-xr-x  1 root  wheel    342 Sep 18  2009 pkginfo*
[...]
lrwxr-xr-x  1 root  wheel      9 Jul 15 01:58 python@ -> python2.6
lrwxr-xr-x  1 root  wheel     16 Jul 15 01:58 python-config@ -> python2.6-config
-rwxr-xr-x  2 root  wheel  24932 Jul 15 01:57 python2.6*
-rwxr-xr-x  2 root  wheel   1473 Jul 15 01:57 python2.6-config*
[...]

If you install 2.7 or 3.1 Pythons using python.org installers or MacPorts, they are all rooted under /Library/Frameworks or /opt/local/Library/Frameworks and will happily co-exist with other versions. To successfully use framework installs, you need to manage your shell PATH. In particular, you should ensure that the framework bin directory of the Python(s) you want to use are on your PATH. So for MacPorts, something like this in ~/.bash_profile:

export PATH="/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin:/opt/local/bin:$PATH

or for the python.org installers:

export PATH="/Library/Frameworks/Python.framework/Versions/2.6/bin:$PATH

In addition, generally you need to install a separate version of your favorite Python package management add-ons (like pip or the easy_install command from Distribute or setuptools) for each Python instance you use. Each one is installed into the default script directory (i.e. the framework bin directory). MacPorts takes care of that automatically when you install its version of the package (sudo port install py26-pip or sudo port install py31-distribute). Note that Apple supplies a set of easy_install links in /usr/bin for its Pythons and they install packages to /Library/Python. You need to make sure you use the easy_install or pip appropriate for your instance of Python. Making sure the instance's framework bin directory is first in the PATH ensures that.

Note, if you use MacPorts you can use its python_select command to manage which Python instance is pointed to by /opt/local/bin/python. Then if /opt/local/bin appears earlier in your PATH list than other directories with a python it will govern which is your default python. But you really need to manage the PATH by adding the framework bin directory so python_select is generally not all that useful.

Another positive aspect of using MacPorts (or similar broad-based distributor) is that it makes available many popular Python packages including handling the occasional messy details of building and linking to 3rd-party C libraries not shipped with OS X. For example, the MySQLdb and PIL packages are frequent stumbling blocks on OS X because of the MySQL client libraries and libjpeg but MacPorts makes it much easier to get everything right. But you do have the choice to install things directly using MacPorts (sudo port py26-pil) or via pip (sudo pip install pil) or easy_install or you can download the source and do it yourself (sudo python setup.py ...). And because all of them under the covers use Python's Distutils to do the dirty work, all of these methods will generally end up installing a package's files in the same place for a given Python instance.