Interfaces and unit tests - always white-box testing?

Question

I have finally got in my mind what worried me about Dependency Injection and similar techniques that should make unit tests easier. Let's take this example:

public interface IRepository { void Item Find(); a lot of other methods here; }
[Test]
public void Test()
{
  var repository = Mock<IRepository>();
  repository.Expect(x => x.Find());
  var service = new Service(repository);
  service.ProcessWithItem();
}

Now, what's wrong with the code above? It's that our test roughly peeks into ProcessWithItem() implementation. What if it wants to do "from x in GetAll() where x..." - but no, our test knows what is going to happen there. And that's just a simple example. Imaging few calls that our test now is tied with, and when we want to change from GetAll() to a better GetAllFastWithoutStuff() inside the method... our test(s) are broken. Please change them. A lot of crappy work that happens so often without any real need.

And that's what often makes me to stop write tests. I just don't see how I can test without knowing implementation details. And knowing them, tests are now very fragile and pain to do.

Sure, it's not about interface (or DI) only. POCOs (and POJOs, why not) also suffer from the same thing, but they're now tied with the data, not with the interface. But the principle is the same - our final assertion is tightly coupled with our knowledge of what our SUT is going to do. "Yes you HAVE to provide this field, sir, and this better be of this value".

As a consequence, tests ARE going to fail - soon and often. This is pain. And the problem.

Are there any techniques to deal with this? AutoMockingContainer (which basically takes care all ALL methods and nested DI hierarchies) looks promising, but with its own drawback. Anything else?

Answer 1

Remember when you're writing a test you're not testing your repository, you're testing your Service class. In this specific example ProcessWithItem method. You create your expectations for repository object. By the way, you forgot to specify expected return for your x.Find method. That's the beauty of DI that you isolate everything from the code you about to write (I assume you do TDD).

To be honest I cannot relate to the problem you describe.

Answer 2

Yeah, that's one of the big problems with unit testing. That, and refactoring. And design changes that are a regular occurrence with Agile. And the inexperience of those creating the tests. And etc etc...

I think the only thing the average non-critical-systems developer can do is pick and choose your battles wisely. Early in development identify the truly critical paths and test those. Weigh the likelihood of that code changing before spending lots of time testing the rest of it.

If anybody figures it all out please let us know.

Answer 3

Dependency Injection, per se, would let you inject an implementation of IRepository that accepts whatever calls are made on it, checks that the invariants and preconditions are satisfied, and returns results satisfying the postconditions. When you choose to inject a mock object that has very specific expectations for what methods will be called, then yes, you're doing highly implementation-specific testing -- but Dependency Injection is totally innocent in the matter, since it never dictates WHAT you should inject; rather, your beef appears to be with Mocking -- in fact, specifically the somewhat-automated mocking approach that you have chosen to use, which is one based on very specific expectations.

Mocking with very specific expectations IS indeed useful for white-box testing only. Depending on the tools / frameworks / libraries you're using (and you're not even specifying the exact programming language in a tag, so I assume your question is totally open ended) you may be able to specify the degrees of freedom allowed (these calls are allowed to come in any orders, these arguments must only satisfy the following preconditions, etc, etc). However, I don't know of an automated tool to perform exactly what you need for opaque-box testing, which is the "generic, tolerant implementation of yonder interface with all the ''programming by contract'' checks that are needed and no other".

What I tend to do over the life of a project is to build up a library of "not quite mocks" for the major interfaces needed. In some cases those may be somewhat obvious from the start, but in other cases they emerge incrementally as I'm considering some major refactoring, as follows (typical scenario)...:

The early stages of the refactoring break some aspect of the fragile strong-expectations mocking that I have cheaply put in place initially, I ponder whether to just tweak the expectations or go whole hog, if I decide it's not a one-off (i.e. the return in future refactorings and tests will justify the investment) then I hand-code a good "not quite mock" and stash it away in the project's specific bag of tricks -- actually often reusable across projects; such classes/packages as MockFilesystem, MockBigtable, MockDom, MockHttpClient, MockHttpServer, etc etc, go into a project-agnostic repository and get reused for testing all kinds of future projects (and in fact may be shared with other teams across the company, if several teams are using filesystem interfaces, bigtable interfaces, DOMs, http client/server interfaces, etc etc, that are uniform across the teams).

I acknowledge that the use of the word "mock" may be slightly inappropriate here if you take "mock" to refer specifically to the precise-expectation style of "fake implementation for testing purposes" of interfaces. Maybe Stub, Shim, Fake, Test, or some other prefix yet might be preferable (I do tend to use Mock for historical reasons, except when I remember to specifically call it Fake or the like;-).

If I was using languages with clear and precise way to express in the language itself the various design-by-contract specs in an interface, I imagine I'd get automatic tool support for most of this faking/shimming/etc; however I mostly code in other languages so I have to do a bit more manual work here. But I think that's a separate issue.

Answer 4

I read the excellent book http://www.manning.com/rainsberger/. I would like to provide some insight I gained from it. I believe several advice could help you to reduce the coupling between your tests and your implementation.

Edited: included in this coupling is the test asserting that the code under test calls some methods. Calling some method is never a functional need, it is an implementation concern. It relates to an interface other than the one being tested.

1. In many cases, the testing should be about the external behavior of an interface, and be completely black-box testing them.

   The author gives the example that the test classes should be in a different package than the class to test. At first, I was sure this was wrong, because it makes it more difficult to test protected and package methods. But he argues that you should only test the external behavior of a system, that is the public methods. The non-public methods are implementation-details, and testing it results in coupling the test with the implementation. This was very insightful to me.

   By the way, this book has so many excellent practical advice on how to design tests (say JUnit tests), that I would buy it on my own money if it wasn't provided by the company! ;-)

2. An excellent other advice from the book was to test at the functionality level, not the method level. For example, testing the add() method for a list requires trusted size() and get() methods, but they in turn require add() so we have a loop, we can't test safely. But testing the list's behavior globally (accross all methods) when adding involves testing the three methods at the same time, not proving that each is correct in isolation, but checking that together they provide the expected behavior. Often, when you try to test one of your methods in isolation, you cannot write a sensible test without using other methods, so you end up testing the implementation instead ; the consequence are coupling between test and implementation.
   Only test functionalities, not methods.

3. Also, note that testing using external ressources (the database being the more common, but many others exist) is much slower, requires some access (IP, licence etc) from the executing machine, require a started container, may be sensitive to simultaneous access (a database can't run reliably multiple JUnit campaign at the same time), and has many other drawbacks. If all your tests use external resources, then you are in trouble, you can't run all your tests all the time, from any machine, from many machines at once, etc. So I understood (still from the book):

   Test only once each external resource (database for example), in a dedicated test that is not a unit-test, but an integration test (although it can still use the same JUnit technology if appropriate).

   Test enough dedicated tests to trust the resource is working. Then, other tests should never test it again, this is a waste, they should trust it.

   Note that the current Maven best-practices give similar advice (see free book "Better builds with Maven"). I believe this is not a coincidence:

   • The JUnits in the test directory of a project are real unit tests. They run every time you do something with your project (except just compile).
   • The integration and functional tests should be provided in a different project, an integration-test project. They only run in a much later (optional) phase, after you have deployed your whole application in the container.

Answer 5

As a consequence, tests ARE going to fail - soon and often. This is pain. And the problem.

Well yes, unit tests can depend on internal implementation details. And sure, such "white box" tests are more brittle than "black box" tests which only rely on the externally published contract.

But I don't agree that this has to cause regular test failures. Think about how you arrived at testing with mocks in the first place: you've used dependency injection to limit the responsibilities of the class, to decrease coupling to other code, and to enable testing the class in isolation.

Are there any techniques to deal with this?

A good unit test can only fail if you change the class under test, even if it depends on internal implementation details. And you can limit the responsibilities and coupling (to other classes) of your class, so that you will rarely have to change it.

In practice you'll have to be pragmatic; every now and then you'll write "unit tests" that are actually integration tests involving multiple classes or over-sized classes. Brittle tests depending on internal implementation details are more dangerous in that case. But for truly TDD-style classes, not so much.