how can I stop all tests from inside a test or setUp using python's unittest framework

Question

I'm extending the python 2.7 unittest framework to do some function testing. One of the things I would like to do is to stop all the tests from running inside of a test, and inside of a setUpClass method. Sometimes if a test fails, the program is so broken it is no longer of any use to keep testing, so I want to stop the tests from running. I noticed that a TestResult has a shouldStop attribute, and a stop() method, but I'm not sure how to get access to that inside of a test. Does anyone have any ideas? Is there a better Way? Thanks for your time.

Answer 1

Currently, you can only stop the tests at the suite level. Once you are in a TestCase, the stop() method for the TestResult is not used when iterating through the tests.

Somewhat related to your question, if you are using python 2.7, you can use the -f/--failfast flag when calling your test with python -m unittest. This will stop the test at the first failure.

See 25.3.2.1. failfast, catch and buffer command line options

You can also consider using Nose to run your tests and use the -x, --stop flag to stop the test early.

Answer 2

I looked at the TestCase class and decided to subclass it. The class just overrides run(). I copied the method and starting at line 318 in the original class added this:

# this is CPython specific. Jython and IronPython may do this differently
if testMethod.func_code.co_argcount == 2:
    testMethod(result)
else:
    testMethod()

It has some CPython specific code in there to tell if the test method can accept another parameter, but since I'm using CPython everywhere, this isn't an issue for me.

Answer 3

In case you are interested here is a simple example how you could make a decision yourself about exiting a test suite cleanly with py.test:

# content of test_module.py
import py
counter = 0
def setup_function(func):
    global counter
    counter += 1
    if counter >=3:
        py.test.exit("decided to stop the test run")

def test_one():
    pass
def test_two():
    pass
def test_three():
    pass

and if you run this you get:

$ py.test test_module.py 
============== test session starts =================
platform linux2 -- Python 2.6.5 -- pytest-1.4.0a1
test path 1: test_module.py

test_module.py ..

!!!! Exit: decided to stop the test run !!!!!!!!!!!!
============= 2 passed in 0.08 seconds =============

You can also put the "py.test.exit()" call inside a test or into a project-specific plugin.

sidenote: py.test natively supports "py.test --maxfail=NUM" to implement stopping after NUM failures. sidenote2: py.test only has limited support for running traditional unittest.TestCase style tests.