The Agile Way: Integration Testing vs Functional Testing or both?

Question

Hi,

I would in an office which has been doing Agile for a while now. We use Scrum for project management and mix in the engineering practices of XP. It works well and we are constantly learning lessons and refining our processes.

I would just like to tell you about our usual practices for testing and get feed back on how this could be improved:

TDD: First Line of Defense We are quite religious about unit testing and I would say our developers are also experienced enough to write comprehensive tests and always isolate the SUT with mocks

Integration Tests For use integration tests are basically the same as the unit tests just without using the mocks, this tends to catch a few issues which slipped through the unit tests. These tests tend to be difficult to read as they usually involve a lot or work in the before_each and after_each sections of the spec framework as the system has to often reach a curtain state in order for the tests to be meaningful.

Functional Testing We usually do this in a structured, but manual fashion. We have played with Selenium and Windmill which are cool, but for us at least not quite there yet.

I would like to hear how anyone else is doing things. Do you think that if Integration Tests or Functional Testing are being done well enough the other can be disregarded?

Regards,

Chris

Answer 1

Unit testing, integration testing and functional testing all serve different purposes. You should not discard one just because the others are running at a high level of reliability.

Answer 2

I would say (and this is just a matter of opinion) that your Functional tests are your true tests. Ie those tests that actually simulate real-life usage of your application. For this reason, never get rid of them, no matter what.

It sounds like you have a decent system going. Keep it all if you have nothing to lose.

Answer 3

We have experienced that a solid set of selenium tests actually sums up what the customer expects of quality really well. So, in essence we've been having this discussion: If writing selenium tests was as easy as writing unit tests we should focus less on unit tests.

And if there is a bug somewhere that does not have any consequence in the application, who really cares? But there's always the issues surrounding real-life complexities; are you sure your functional tests are capturing the correct scenarios ? There may be underlying complexities caused by other systems that are not directly visible in the behavior of the application.

In reality, writing selenium tests (using a proper programming language, not selenese) does get really simple and fun once you drill through the initial problems. But we're not willing to give up our unit tests quite yet....

Answer 4

Unit, integration and functional testing, though exercising the same code, are attacking it from different perspectives. It's those perspectives that make the difference, if you were to drop one type of testing then something could work it's way in from that angle.

Also, unit testing isn't really about testing your code, especially if you are practising TDD. The process of TDD helps you design your code better, you just get the added bonus of a suite of tests at the end of it.

You haven't mentioned whether you have a continuous integration server running. I would strongly recommend setting one up (Hudson is easy to set up). Then you can have your integration and functional tests run against every check in of the code.

Answer 5

I tend not to separate various flavours of testing in TDD. For me TDD is Test-Driven Development, not Unit Test-Driven Development. So my TDD practice combines unit tests, integration tests, functional and acceptance tests. This results in some components being covered by certain types of tests and others components being covered by other types of tests in a very pragmatic fashion.

I have asked a question about the relevance of this practice and the short answer was that in practice the separation is between fast/simple tests run automatically at every build and slow/complex tests run less often.

Answer 6

My company does functional testing but not unit or integration testing. Im trying to encourage we adopt them, i see them as encouraging better design and a faster indication that all is well. Do you need unit tests if you do functional testing?

Answer 7

At my current client we don't really separate between unit-tests and integration-tests. The business entities are so dependent on the underlying data-layer (using a homegrown ORM framework) that in effect we have little or no true unit-tests.

The build server is set up with continous integration (CI) in Team Build and to keep this from bogging too much with slow tests (the full test suite takes over an hour to run on the server) we have separated the tests into "slow" tests that gets run twice a day and "fast" tests that get run as part of continous integration. By setting an attribute on the test-method the build-server can tell the difference between the two.

In general, "slow" tests are any that needs to do data-access, use web-services or similar. These would be considered integration tests or functional tests by common convention. Examples are: CRUD-tests, business validation rule tests that need a set of objects to work with, etc.

"Fast" tests are more like unit-tests, where you can reasonably isolate a single object's state and behavior for the test.

I would consider any test that run in tenths of a second or less to be "fast". Anything else is slow and probably shouldn't be run as part of CI.

I agree that you should not get too hung up on the "flavor" of test you use as part of development (expressing acceptance criteria as tests is the exception of course). The individual developer should use their judgement in deciding what type of tests best suit their code. Insisting on unit-tests for a business entity might not reveal the faults a CRUD-test would and so on...

Answer 8

I really like Gojko Adzic's concept of a "face saving test" as a way to determine what you test via the UI: http://gojko.net/2007/09/25/effective-user-interface-testing/

Answer 9

You should do all of them because unit,integration and functional testing all serve different purposes.
Belongs to me an important point is that the way write test is very important and TDD is not enought, BDD (behavior driven development) give a good approach...