Python, unit-testing and mocking imports

Question

Hello,

I am in a project where we are starting refactoring some massive code base. One problem that immediately sprang up is that each file imports a lot of other files. How do I in an elegant way mock this in my unit test without having to alter the actual code so I can start to write unit-tests?

As an example: The file with the functions I want to test, imports ten other files which is part of our software and not python core libs.

I want to be able to run the unit tests as separately as possible and for now I am only going to test functions that does not depend on things from the files that are being imported.

Thanks for all the answers.

I didn't really know what I wanted to do from the start but now I think I know.

Problem was that some imports was only possible when the whole application was running because of some third-party auto-magic. So I had to make some stubs for these modules in a directory which I pointed out with sys.path

Now I can import the file which contains the functions I want to write tests for in my unit-test file without complaints about missing modules.

Answer 1

"imports a lot of other files"? Imports a lot of other files that are part of your customized code base? Or imports a lot of other files that are part of the Python distribution? Or imports a lot of other open source project files?

If your imports don't work, you have a "simple" PYTHONPATH problem. Get all of your various project directories onto a PYTHONPATH that you can use for testing. We have a rather complex path, in Windows we manage it like this

@set Part1=c:\blah\blah\blah
@set Part2=c:\some\other\path
@set that=g:\shared\stuff
set PYTHONPATH=%part1%;%part2%;%that%

We keep each piece of the path separate so that we (a) know where things come from and (b) can manage change when we move things around.

Since the PYTHONPATH is searched in order, we can control what gets used by adjusting the order on the path.

Once you have "everything", it becomes a question of trust.

Either

• you trust something (i.e., the Python code base) and just import it.

Or

• You don't trust something (i.e., your own code) and you

  1. test it separately and
  2. mock it for stand-alone testing.

Would you test the Python libraries? If so, you've got a lot of work. If not, then, you should perhaps only mock out the things you're actually going to test.

Answer 2

If you really want to muck around with the python import mechanism, take a look at the ihooks module. It provides tools for changing the behavior of the __import__ built-in. But it's not clear from your question why you need to do this.

Answer 3

No difficult manipulation is necessary if you want a quick-and-dirty fix before your unit-tests.

If the unit tests are in the same file as the code you wish to test, simply delete unwanted module from the globals() dictionary.

Here is a rather lengthy example: suppose you have a module impp.py with contents:

value = 5

Now, in your test file you can write:

>>> import impp
>>> print globals().keys()
>>> def printVal():
>>>     print impp.value
['printVal', '__builtins__', '__file__', 'impp', '__name__', '__doc__']

Note that impp is among the globals, because it was imported. Calling the function printVal that uses impp module still works:

>>> printVal()
5

But now, if you remove impp key from globals()...

>>> del globals()['impp']
>>> print globals().keys()
['printVal', '__builtins__', '__file__', '__name__', '__doc__']

...and try to call printVal(), you'll get:

>>> printVal()
Traceback (most recent call last):
  File "test_imp.py", line 13, in <module>
    printVal()
  File "test_imp.py", line 5, in printVal
    print impp.value
NameError: global name 'impp' is not defined

...which is probably exactly what you're trying to achieve.

To use it in your unit-tests, you can delete the globals just before running the test suite, e.g. in __main__:

if __name__ == '__main__':
    del globals()['impp']
    unittest.main()

Answer 4

In your comment above, you say you want to convince python that certain modules have already been imported. This still seems like a strange goal, but if that's really what you want to do, in principle you can sneak around behind the import mechanism's back, and change sys.modules. Not sure how this'd work for package imports, but should be fine for absolute imports.

Answer 5

If you want to import a module while at the same time ensuring that it doesn't import anything, you can replace the __import__ builtin function.

For example, use this class:

class ImportWrapper(object):
    def __init__(self, real_import):
        self.real_import = real_import

    def wrapper(self, wantedModules):
        def inner(moduleName, *args, **kwargs):
            if moduleName in wantedModules:
                print "IMPORTING MODULE", moduleName
                self.real_import(*args, **kwargs)
            else:
                print "NOT IMPORTING MODULE", moduleName
        return inner

    def mock_import(self, moduleName, wantedModules):
        __builtins__.__import__ = self.wrapper(wantedModules)
        __import__(moduleName, globals(), locals(), [], -1)
        __builtins__.__import__ = self.real_import

And in your test code, instead of writing import myModule, write:

wrapper = ImportWrapper(__import__)
wrapper.mock_import('myModule', [])

The second argument to mock_import is a list of module names you do want to import in inner module.

This example can be modified further to e.g. import other module than desired instead of just not importing it, or even mocking the module object with some custom object of your own.