setuptools test hides import errors. How to have better info ?

Question

In python setuptools, python setup.py test runs the testsuite. However if I have an import error in my testsuite, the only error message I obtain is an AttributeError complaining that my test class is missing. Is there a way to obtain a more detailed error message, so I can fix the testsuite ?

I will explain myself better with the following example. Suppose I have a package called foo, created anew with paster. I then add the test

./foo
./foo/__init__.py
./foo/tests
./foo/tests/__init__.py
./foo/tests/mytest.py
./setup.cfg
./setup.py

Now, suppose mytest.py contains the following code

import unittest
class MyTestClass(unittest.TestCase):
    def testFoo(self):
        self.assertEqual(1,1)

This works. However, if I try to import an unexistent module

import unittest
import frombiz
class MyTestClass(unittest.TestCase):
    def testFoo(self):
        self.assertEqual(1,1)

This is the error I obtain

Traceback (most recent call last):
  File "setup.py", line 26, in <module>
    test_suite = "foo.tests"
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/command/test.py", line 121, in run
    self.with_project_on_sys_path(self.run_tests)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/command/test.py", line 101, in with_project_on_sys_path
    func()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/command/test.py", line 130, in run_tests
    testLoader = loader_class()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 816, in __init__
    self.parseArgs(argv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 843, in parseArgs
    self.createTests()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 849, in createTests
    self.module)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 613, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 587, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/command/test.py", line 34, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 584, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'mytest'

In other words, there's no reference to the failed import.

Answer 1

The problem is that the first argument of __import__() must be module. When you need to reach some object with dotted name you never know what part is module and what isn't. The one way to get module.subname object is to try import it as submodule first and when it fails use getattr(module, subname), as unittest does. This will sometimes give you AttributeError instead of ImportError. Another way to do this is to try getattr(module, subname) first and only when it fails try import. This way isn't better: it will sometimes give you ImportError when more appropriate would be AttributeError.

Probably the best unittest can do in this case is to raise its own exception saying that both import and attribute lookup failed. Try to send bug report for this problem.

Answer 2

You may also want to try your code with Distribute (a fork of setuptools since setuptools isn't being very actively maintained). I don't know if it will do anything different, but it's worth a shot.

Answer 3

The issue here is that a Python ImportError doesn't tell you what module the error actually occurred in, unless you carefully inspect the traceback -- which the unittest module doesn't do. You'll have the same problem with the unittest module, regardless of what tool you use to run it.

You may want to try the "nose" package - it has a setuptools plugin, so you can add it to your setup.py and let it find and import your tests, instead of using setuptools' default test finder. If I recall correctly, it loads test code in a different way than unittest does, and it may be able to give a better error message in this case. And even if it doesn't, it'd probably be easier to get it added to nose than to unittest!