What is the right balance between unit vs. functional testing in a typical web application?

Question

Unit tests are cheaper to write and maintain, but they don't cover all scenarios. What is the right balance between them?

Answer 1

On the app I am working on at the moment, there are probably 10:1 unit to functional tests. The unit test work simply things like retrieving entities from DB, error handling tests for db/network connectivity etc. These things run quick - minutes or less and are run by devs daily.

functional tests while fewer tend to be kitchen sink approach - can user complete order etc. tend to cover the business domain end of things are run by business analsyts and operations - sadly for us, often by hand. These things take weeks to run and usually to finalize a release cycle.

Answer 2

it is important to distinguish between the intent and scope of these two types of tests:

• a unit test typically test a specific feature at the module/class level, e.g. create-X, update-Y, foo-the-bar, compact-the-whizbang, etc. One class may have multiple unit tests

• a functional test, also called an 'acceptance test' typically tests a use-case scenario from the outermost interface through to the end of processing, e.g. from the user-interface to the database and back again, from the input process to the notification utility, etc.

these two types of tests are not interchangable, and are in general disjoint. So the notion of striking a 'balance' between them makes no sense. You either need them or you don't.

if you are referring to the ease of coding each type of test in your testing framework, that is a different question - but the use of the framework (say, NUnit vs. a user-bot) does not change the type/nature of the test.

the best "balance", in general, would be to unit-test for confidence and completeness, and functional-test for client acceptance

Answer 3

My current project is at about 60% unit test coverage and all user stories have happy-day coverage of user stories in selenium tests, some have additional coverage.

We're continually discussing this: Is there really any point of pushing unit-test coverage of increasingly absurd scenarios for much higher coverage ?

The argument is that expanding selenium tests increases test coverage on things that have business value. Does the customer really care about unit test corner cases that may fail?

When you get good at selenium testing, the cost of writing new tests with business value decreases. For us they're just as simple as unit tests.

The run-time cost is another issue. We have a small cluster of boxes running these tests all of the time.

So we tends to favour web-tests more, probably because we've gotten good at writing them and they provide undeniable business value.

Answer 4

Originally I leaned heavily towards preferring unit tests over functional/acceptance tests due to the initial cost factor of acceptance tests. However, over time I have changed my philosophy and am now a strong proponent of choosing acceptance tests wherever possible and only using unit tests when acceptance tests can't meet my needs.

The basic rational behind choosing acceptance over unit tests is the same as the basic rational for SOLID code. Your implementation should be able to change drastically with refactoring etc, but all business cases - acceptance tests- should be able to remain un-changed and prove acceptable system behavior (tests pass). With unit tests there's often a natural strong coupling of test to implementation code. Even though it's test code, it's still code and strong coupling as-we-know should be avoided. By choosing acceptance tests you're often led down the spiral of success to create well-planned, consumable de-coupled api's letting your implementation change behind the scenes without forcing your tests to change. Also, your developer implementation thoughts are in-line with the business system-behavior thoughts. In the end I find that all of this is better for business and for coder satisfaction.

From a theory standpoint I often ask my-self if a piece of code can't be tested via an acceptance test why that piece of code should exist? IE - if it's not part of a valid business scenario, then does that code add value or is it currently and will it remain solely cost?

Additionally, if you comment/document your acceptance tests well, those comments/documents generally are the most current and accurate language of the system - which usually will let you avoid other less valuable documentation approaches. Unit tests don't lend themselves to that form of "business-term" communication.

Lastly, I haven't formed this view point from just my personal development, it's proven successful with a couple different project teams in my "very corporate" work environment.

JB http://jb-brown.blogspot.com

Answer 5

Tests should run quickly and help localise problems. Unit tests allow you to do this by only testing the module in question. However functional/integration/acceptance tests can be made to run sufficiently quickly in most web scenarios.

Answer 6

I like Brian Marick's quadrant on automated tests where the distinctions are business vs. technology facing and support programming vs. critique product.

With that framework the question of balance becomes, what do I need right now?

Answer 7

I agree with Steven Lowe that there is no trade-off between unit testing and functional testing, as they are used for very different purposes.

Unit tests are about method and type verification, and also regression testing. Functional tests are about functional, scenario, and feasibility testing. In my opinion, there is almost no overlap,

If it helps, here are my testing categories.

Developers start from the inside and work outwards, focusing on code:

• Assertions - verify data flow and structures
• Debugger - verify code flow and data
• Unit testing - verify each function
• Integration testing - verify sub-systems
• System testing - verify functionality
• Regression tests - verify defects stay fixed
• Security tests - verify system can't be penetrated easily.

Testers start from the outside and work inwards, focusing on features:

• Acceptance tests - verify end-user requirements
• Scenario tests - verify real-world situations
• Global tests - verify feasible inputs
• Regression tests - verify defects stay fixed
• Usability tests - verify that system is easy to use
• Security tests - verify system can't be penetrated easily
• Code coverage - testing untouched code
• Compatibility - with previous releases
• Looking for quirks and rough edges.

End-users work from the outside, and usually have little focus:

• Acceptance tests - verify end-user requirements
• Scenario tests - verify real-world situations
• Usability tests - verify that system is easy to use
• Looking for quirks and rough edges.