How to distribute and execute platform-specific unit tests?

Question

We have a python project that we want to start testing using buildbot. Its unit tests include tests that should only work on some platforms. So, we've got tests that should pass on all platforms, tests that should only run on 1 specific platform, tests that should pass on platforms A, B, C and tests that pass on B and D.

What is the best way of doing this? Simple suites would be a hassle, since, as described, each test can have a different list of target platforms. I thought about adding "@run_on" and "@ignore_on" decorators that would match platforms to test methods. Is there anything better?

Answer 1

Sounds like a handy place for a test loader.

Check out http://docs.python.org/library/unittest.html#unittest.TestLoader.loadTestsFromName

If you provide some suitable naming conventions you can probably create suites based on your test naming conventions.

If I've got test A that executes on AIX, Linux (all) and 32 bit Windows, test B that runs on Windows 64, Linux 64 and Solaris, and test C that runs on everything but HPUX and test D that runs on everything... What possible naming-convention is there for this?

class TestA_AIX_Linux2_Win32( unittest.TestCase ):

class TestB_Win64_Linux64_Solaris( unittest.TestCase ):

class TestC_AIX_Linux2_Win32_Win64_Linux64_Solaris( unittest.TestCase ):

class TestD_All( unittest.TestCase ):

The hard part is the "not HP/UX". Avoiding negative logic makes your life simpler. In this case, you simply list all OS's that are not HP/UX. The list is fairly short and grows slowly.

The "All" tests are simply a separate text search that's merged in with the current platform's list of tests to create a complete suite.

You could try something like

class TextC_XHPUX( unittest.TestCase ):

Your text matching rule is normally "_someOSName"; your exceptions would be a weird text filter to mess around with the the "_X" name.

"We can't have a positive list of OS's. What if we add a new OS? Do we have to rename every test to explicitly include it?" Yes. The new operating system market is slow to evolve, it's not that painful to manage.

The alternative is to include information within each class (i.e., a class-level function) or a decorator and use a customized class loader that evaluates the class-level function.

Answer 2

On a couple of occasions I have used this very simple approach in test modules:

import sys
import unittest

if 'win' in sys.platform:
    class TestIt(unittest.TestCase):
        ...

if 'linux' in sys.platform:
    class TestIt(unittest.TestCase):
        ...

Answer 3

We've decided to go with decorators that, using platform module and others, check whether the tests should be executed, and if not simply let it pass (though, we saw that python2.7 already has in its trunk a SkipTest exception that could be raised in such cases, to ignore the test).