How do I override a python import?

Question

So I'm working on pypreprocessor which is a preprocessor that takes c-style directives and I've been able to make it work like a traditional preprocessor (it's self-consuming and executes postprocessed code on-the-fly) except that it breaks library imports.

The problem is. The preprocessor runs through the file, processes' it, outputs to a temp file, and exec() the temp file. Libraries that are imported need to be handled a little different because they aren't executed but rather loaded and made accessible to the caller module.

What I need to be able to do is. Interrupt the import (since the preprocessor is being run in the middle of the import), load the postprocessed code as a tempModule, and replace the original import with the tempModule to trick the calling script with the import into believing that the tempModule is the original module.

I have searched everywhere and so far, have no solution.

This question is the closest I've seen so far to providing an answer: http://stackoverflow.com/questions/1096216/override-namespace-in-python

Here's what I have.

# remove the bytecode file created by the first import
os.remove(moduleName + '.pyc')
# remove the first import
del sys.modules[moduleName]
# import the postprocessed module
tmpModule = __import__(tmpModuleName)
# set first module's reference to point to the preprocessed module
sys.modules[moduleName] = tmpModule

moduleName is the name of the original module, tmpModuleName is the name of the postprocessed code file.

The strange part is, this solution still runs completely normal as if the first module completed loaded normally; unless you remove the last line, then you get a module not found error.

Hopefully someone on SO know a lot more about imports than I do because this one has me stumped.

Note: I will only award a solution, or, if this is not possible in python; the best, most detailed explanation of why this is not impossible.

Update: For anybody who is interested here is the working code.

if imp.lock_held() is True:
    del sys.modules[moduleName]
    sys.modules[tmpModuleName] = __import__(tmpModuleName)
    sys.modules[moduleName] = __import__(tmpModuleName)

The 'imp.lock_held' part detects whether the module is being loaded as a library. The following lines do the rest.

Answer 1

Very strange approach -- emulate low level language with high-level one. If first simple way don't work, may be its a wrong goal?

BTW, C preprocessor simply operates with text files and a set of preprocessor variables, not high-level loaded/unloaded modules.

Answer 2

Does this answer your question? The second import does the trick.

Mod_1.py

def test_function():
    print "Test Function -- Mod 1"

Mod_2.py

def test_function():
    print "Test Function -- Mod 2"

Test.py

#!/usr/bin/python

import sys

import Mod_1

Mod_1.test_function()

del sys.modules['Mod_1']

sys.modules['Mod_1'] = __import__('Mod_2')

import Mod_1

Mod_1.test_function()

Answer 3

To define a different import behavior or to totally subvert import process you will need to write import hooks see PEP 302

e.g.

import sys

class MyImporter(object):

    def find_module(self, module_name, package_path):
        # return a loader
        return self

    def load_module(self, module_name):
        # return a module
        return self

sys.meta_path.append(MyImporter())

import now_you_can_import_any_name
print now_you_can_import_any_name

It outputs:

<__main__.MyImporter object at 0x009F85F0>

So what it basically does it returns a new module (which can be any object) in this case itself, you may use it to alter import behavior by returning processe_xxx on import of xxx

IMO: python doesn't need a preprocessor whatever you are accomplishing can be accomplished in python itself due to it very dynamic nature, e.g taking the case of debug example, what is wrong with having at top of file

debug = 1

and later

if debug:
   print "wow"

Answer 4

In Python 2 there is the imputil module that seems to provide the functionality you are looking for, but has been removed in python 3. It's not very well documented but contains an example section that shows how you can replace the standard import functions.

For Python 3 there is the importlib module (introduced in Python 3.1) that contains functions and classes to modify the import functionality in all kinds of ways. It should be suitable to hook your preprocessor into the import system.