How To Run integrational Tests

Question

In our project we have a plenty of Unit Tests. They help to keep project rather well-tested.

Besides them we have a set of tests which are unit tests, but depends on some kind of external resource. We call them external tests. They can access web-service sometimes or similar.

While unit tests is easy to run the integrational tests couldn't pass sometimes - for example due to timeout error. Also these tests can take too much time to run.

Currently we keep integration/external unit tests just to run them when developing corresponding functionality.

For plain unit tests we use TeamCIty for continuous integration.

How do you run the integration unit tests and when do you run them?

Answer 1

We run all the tests in one huge suite. It takes 7 minutes to run.

Our integration tests create mock servers. They never time out -- except when the test requires the server to time out.

So we have the following kinds of things. (The code sample is Python)

class SomeIntegrationTest( unittest.TestCase ):
    def setUp( self ):
        testclient.StartVendorMockServer( 18000 ) # port number
        self.connection = applicationLibrary.connect( 'localhost', 18000 )
    def test_should_do_this( self ):
        self.connection.this()
        self.assert...
    def tearDown( self ):
        testClient.KillVendorMockServer( 18000 )

This has some limitations -- it's always forking the client mock server for each test. Sometimes that's okay, and sometimes that's too much starting and stopping.

We also have the following kinds of things

class SomeIntegrationTest( unittest.TestCase ):
    def setUp( self ):
        self.connection = applicationLibrary.connect( 'localhost', 18000 )
    def test_should_do_this( self ):
        self.connection.this()
        self.assert...

 if __name__ == "__main__":
     testclient.StartVendorMockServer( 18000 ) # port number
     result= unittest.TextTestRunner().run()
     testclient.KillVendorMockServer( 18000 )
     system.exit( result.failures + result.errors )

To support this testing, we have a number of mocked-up servers for various kinds of integration tests.

Answer 2

We're using Maven2: maven-surefire-plugin to run unit tests (in the test phase) and maven-failsafe-plugin for integration tests (integration-test phase).

By default, all tests run when the project is built, however integration tests can be turned off using profiles.

In many cases integration tests are the part of the module, n some cases there are also dedicated modules which only do integration tests.

One of the teams also uses Fitnesse for acceptance testing. These tests are also in dedicated modules.

We're using Hudson for CI.

Answer 3

In our project we have separate suite for regular/plain unit tests and separate suite for integration tests. The are two reasons for that:

1. performance: integration tests are much slower,
2. test fragility: integration tests fail more often due to environment-related conditions (give false positives).

We use TeamCity as our main Continuous Integration server and Maven as build system. We use the following algorithm to run the tests:

1. We run unit tests at within Eclipse IDE and before every commit.
2. We run unit tests automatically after each commit on TeamCity agents using Maven's mvn clean install
3. We run integration tests automatically on TeamCity agent after "main" build is completed.

The way we trigger integration tests execution is by configuring TeamCity's integration.tests task to be dependent on "main" continous.build task, see here for details: http://confluence.jetbrains.net/display/TCD4/Dependencies+Triggers

We run only integration tests (excluding unit tests) by:

• using separate directory named "src/it/java" to keep integration tests,
• excluding by default this source folder from maven-surefire-plugin configuration (configuration/excludes element),
• using Maven profile called "integration" to exclude regular unit tests and include tests from "src/it/java" (this profile is configured by passing -Pintegration in integration.tests task).