Is it reasonable to enforce that unit tests should never talk to a live database/webservice?

Question

My team continues to find more and more value in the unit tests we write. We don't generally unit test the data access layers of our applications, as they don't contain "logic". In my experience we've run into significant performance issues and non-reproducible errors as a result of letting developers write unit tests that talk to live databases (or webservices) and as a result more and more developers are creating mocks that feed data to these unit tests.

Taking this approach has increased the speed of the tests and isolated testing to the logic rather than testing the connection/retrieval at the same time. I'm wondering if this sounds reasonable to enforce as a coding standard. What are the pros/cons regarding unit testing live databases/webservices that I'm missing?

Answer 1

I generally view a unit test as something that tests an individual class or module in isolation. As such, I try to avoid having unit tests know about live systems or external resources - I tend to avoid such dependencies in unit tests - or mock them out.

Integration tests are a different story. An integration test should avoid mocking out resources, and therefore may indeed require a live database or service to support it. What you are describing sounds like it may be an integration test rather than a unit test.

In some cases, it's desirable to simulate the services that are required for an integration test - but when possible, I try to avoid this because I don't trust the simulation being able to reflect the behavior of the real thing. For me, an integration test should actually test the integration of components in a system.

Answer 2

IMO You are probably missing that if you are using databases/WS on the test, this is not a unit test, but an integration test.

Answer 3

The database and webservice parts of your application should be tested as well, but by definition they will not be unit tests, but integration tests. These tests would be separate from your unit tests and run less often, but will provide very valuable early detection of defects.

Answer 4

Any build validation test should be as complete as possible. If you have a function call that relays on external resources, you must insure that entire system is behaving properly.

The problem with specifically unit tests for functions that connect to a database can be problematic because these functions can have side effects, such as insert data. These side effects may cause return values to change.

Answer 5

In my experience, testing should be done on multiple levels:

• unit tests that check the correctness of classes/modules without the need for external data
• system tests that check bigger parts of the application, loading data from specific databases/files and verifying the results (e.g. by comparing output files). Be sure not to change these databases/files and be sure to have one database/fileset per group of tests
• manual tests that check the user interface

Each of these tests have their usefulness.

Answer 6

I'm sorry, but many developers miss the fact that you need to unit test the database as well. I can agree with mocking the connection between class "A" and the database, but at some point, you're going to create a class that uses some data access technology (ADO.NET, for instance), and you need to test that it actually works.

Now, by keeping these tests small and focused, you can more easily prevent side effects, and ensure the correct initial conditions of the database. But the fact is that you must test such code, or your properly tested business logic will wind up with bad data to work with.

Answer 7

Unit tests shouldn't be done with live data, period.

Even if you want to start doing integration tests, don't start that with live data either.

Beginning integration testing should always be done against either dev data or mockup data. Preferred method would be to have a dev system that has publishes from live data, or use sql scripts to generate a complete set of data for you to test against.