Python egg found interactively but not in fastcgi

Question

In agreement to this question, and its answer. I added the path of the egg and it worked. However, when I run python interactively and I import flup, it works without any problem or additional path specification. Where is the difference ?

Edit: It appears that while doing fastcgi stuff, the .pth files are not parsed, but this is only a guess. Need more official statement.

Answer 1

Programs run by or code run in a web server has a restricted environment compared with what you use interactively. Most likely, the difference stems from the difference between your interactive environment and the FastCGI environment. What I can't tell you is which difference is critical in this context.

Answer 2

I've encountered a similar issue when running my Python applications under IIS (Windows). I've found that when running under ISAPI, eggs aren't readable because setuptools munges the permissions on zipped eggs, and because the ISAPI application runs under a limited privilege account.

You may be experiencing the same situation in FastCGI. If the FastCGI process doesn't have permission to read the eggs or expand the eggs as necessary, you may have problems. Also, I've found that setting the PYTHON_EGG_CACHE environment variable to a directory that's writable by the process is also necessary for some eggs (in particular, eggs with binary/extension modules or resources that must be accessed as files).

Answer 3

After some more thorough analysis, I think I understand what's going on here.

When Python starts up, it sets up the sys.path (all as part of initializing the interpreter).

At this time, the environment is used to determine where to find .pth files. If no PYTHONPATH is defined at this time, then it won't find your modules installed to sys.prefix. Additionally, since easy-install.pth is probably installed to your custom prefix directory, it won't find that .pth file to be parsed.

Adding environment variables to os.environ or sys.path after the interpreter has initialized won't cause .pth files to be parsed again. This is why you're finding yourself forced to manually do what you expect Python to do naturally.

I think the correct solution is to make sure the custom path is available to the Python interpreter at the time the interpreter starts up (which is before mysite.fcgi is executed).

I looked for options to add a PYTHONPATH environment variable to mod_fastcgi, but I see no such option. Perhaps it's a general Apache option, so not documented in mod_fastcgi, or perhaps it's not possible to set a static variable in the mod_fastcgi config.

Given that, I believe you could produce a workaround with the following:

1. Create a wrapper script (a shell script or another Python script) that will be your new FastCGI handler.
2. In this wrapper script, set the PYTHONPATH environment variable to your prefix path (just as you have set in your user environment which works).
3. Have the wrapper script launch the Python process for your original fastcgi handler (mysite.fcgi) with the altered environment.

Although I don't have a good environment in which to test, I think a wrapper shell script would look something like:

#!/bin/sh
export PYTHONPATH=/your/local/python/path
/path/to/python /path/to/your/fastcgi/handler  # this line should be similar to what was supplied to mod_fastcgi originally

There may be alternative workarounds to consider.

• From mysite.fgci, cause the Python to re-process the sys.path based on a new, altered environment. I don't know how this would be done, but this might be cleaner than having a wrapper script.
• Find an Apache/mod_fastcgi option that allows an environment variable (PYTHONPATH) to be specified to the fastcgi process.

Answer 4

I concur that even with the PYTHONPATH environment variable defined the .pth files do not get interpreted when python starts up in the FastCGI environment. I do not now why this is so, but I do have a suggestion for a workaround.

Use site.addsitedir. It will interpret the .pth files allowing you to then import eggs simply by name without having to add the full path to each one.

#!/user/bin/python2.6

import site

# adds a directory to sys.path and processes its .pth files
site.addsitedir('/home/mhanney/.local/lib/python2.6/site-packages/')

# avoids permissions error writing to system egg-cache
os.environ['PYTHON_EGG_CACHE'] = '/home/mhanney/.local/egg-cache'

It is not necessary to use virtual env. At my shared hosting provider I just install eggs in ~/.local using

python setup.py install --prefix=~/.local

Here is a variation on the flup 'Hello World' example to dump the environment vars, path and modules, useful for debugging FastCGI.

#!/usr/bin/python2.6
import sys, os, site, StringIO
from pprint import pprint as p

# adds a directory to sys.path and processes its .pth files
site.addsitedir('/home/mhanney/.local/lib/python2.6/site-packages/')

# avoids permissions error writing to system egg-cache
os.environ['PYTHON_EGG_CACHE'] = '/home/mhanney/.local/egg-cache'

def test_app(environ, start_response):
    output = StringIO.StringIO()
    output.write("Environment:\n")
    for param in os.environ.keys():
        output.write("%s %s\n" % (param,os.environ[param]))
    output.write("\n\nsys.path:\n")
    p(sys.path, output)
    output.write("\n\nsys.modules:\n")
    p(sys.modules, output)
    start_response('200 OK', [('Content-Type', 'text/plain')])
    yield output.getvalue()

if __name__ == '__main__':
    from flup.server.fcgi import WSGIServer
    WSGIServer(test_app).run()