In Python, how can I efficiently manage references between script files?

Question

I have a fair number of Python scripts that contain reusable code that are used and referenced by other Python scripts. However, these scripts tend to be scattered across different directories and I find it to be somewhat tedious to have to include (most often multiple) calls to sys.path.append on my top-level scripts. I just want to provide the 'import' statements without the additional file references in the same script.

Currently, I have this:

import sys
sys.path.append('..//shared1//reusable_foo')
import Foo
sys.path.append('..//shared2//reusable_bar')
import Bar

My preference would be the following:

import Foo
import Bar

My background is primarily in the .NET platform so I am accustomed to having meta files such as *.csproj, *.vbproj, *.sln, etc. to manage and contain the actual file path references outside of the source files. This allows me to just provide 'using' directives (equivalent to Python's import) without exposing all of the references and allowing for reuse of the path references themselves across multiple scripts.

Does Python have equivalent support for this and, if not, what are some techniques and approaches?

Answer 1

If your reusable files are packaged (that is, they include an __init__.py file) and the path to that package is part of your PYTHONPATH or sys.path then you should be able to do just

import Foo

This question provides a few more details.

(Note: As Jim said, you could also drop your reusable code into your site-packages directory.)

Answer 2

The simple answer is to put your reusable code in your site-packages directory, which is in your sys.path.

You can also extend the search path by adding .pth files somewhere in your path. See http://www.python.org/doc/2.5.2/inst/search-path.html#search-path for more details

Oh, and python 2.6/3.0 adds support for PEP370, Per-user site-packages Directory

Answer 3

You can put the reusable stuff in site-packages. That's completely transparent, since it's in sys.path by default.

You can put someName.pth files in site-packages. These files have the directory in which your actual reusable stuff lives. This is also completely transparent. And doesn't involve the extra step of installing a change in site-packages.

You can put the directory of the reusable stuff on PYTHONPATH. That's a little less transparent, because you have to make sure it's set. Not rocket science, but not completely transparent.

Answer 4

In one project, I wanted to make sure that the user could put python scripts (that could basically be used as plugins) anywhere. My solution was to put the following in the config file for that project:

[server]
PYPATH_APPEND: /home/jason:/usr/share/some_directory

That way, this would add /home/jason and /usr/share/some_directory to the python path at program launch.

Then, it's just a simple matter of splitting the string by the colons and adding those directories to the end of the sys.path. You may want to consider putting a module in the site-packages directory that contains a function to read in that config file and add those directories to the sys.path (unfortunately, I don't have time at the moment to write an example).

As others have mentioned, it's a good idea to put as much in site-packages as possible and also using .pth files. But this can be a good idea if you have a script that needs to import a bunch of stuff that's not in site-packages that you wouldn't want to import from other scripts.

(there may also be a way to do this using .pth files, but I like being able to manipulate the python path in the same place as I put the rest of my configuration info)

Answer 5

The simplest way is to set (or add to) PYTHONPATH, and put (or symlink) your modules and packages into a path contained in PYTHONPATH.

Answer 6

My solution was to package up one utility that would import the module: my_util is in site packages

import my_util

foo = myutil.import_script('..//shared1//reusable_foo')
if foo == None:
    sys.exit(1)


def import_script(script_path, log_status = True):
    """
    imports a module and returns the handle
    """
    lpath = os.path.split(script_path)

    if lpath[1] == '':
        log('Error in script "%s" in import_script' % (script_path))
        return None


    #check if path is already in sys.path so we don't repeat
    npath = None
    if lpath[0] == '':
        npath = '.'
    else:
        if lpath[0] not in sys.path:
            npath = lpath[0]

    if npath != None:
        try:
            sys.path.append(npath)
        except:
            if log_status == True:
                log('Error adding path "%s" in import_script' % npath)
            return None

    try:   
        mod =  __import__(lpath[1])
    except:
        error_trace,error_reason =  FormatExceptionInfo()
        if log_status == True:
            log('Error importing "%s" module in import_script: %s' % (script_path, error_trace + error_reason))
        sys.path.remove(npath)
        return None

    return mod