There are some things you just can't test!?

Question

When unit testing, it is always hard to know how low down to the framework you go.

If we have a class that directly depends on the .NET Framework, i.e. the System.IO.File class, then we cant really test that in isolation.

Sure we could wrap it and inject it into the dependent class, but then we start to wrap every .NET class! What if that wrapper is not a straight call through? Perhaps it does some checking/business logic first, that you would want to test?

Currently we wrap down to a certain level, and then just don't bother unit testing that wrapper. Perhaps this is ok, because it will be tested later on through integration and exploratory testing?

Here is an example in C# just to illustrate my point:

This class is tightly coupled to the .NET Framework.. thats fine, but now I cant test it in isolation, it requires files to exist etc.

public class PathResolver
{
    public string Resolve(string filename)
    {
        string completePath = string.Empty;
        if(!File.Exists(filename))
        {
            return Path.Combine(@"D:\MyExample", filename);
        }
        return completePath;
    }
}

We can unit test this by doing something like:

public class PathResolver
{
    private readonly IFileSystem _fileSystem;

    public PathResolver(IFileSystem fileSystem)
    {
        _fileSystem = fileSystem;
    }

    public string Resolve(string filename)
    {
        string completePath = string.Empty;
        if(!_fileSystem.Exists(filename))
        {
            return _fileSystem.Combine(@"D:\MyExample", filename);
        }
        return completePath;
    }
}

But now we cant test the "FileSystem" class!

What are other peoples thoughts?

Answer 1

In general, testing frameworks like .NET is not your responsibility, it's the responsibility of the people releasing the framework.

Answer 2

Do you want to test System.IO ? If not, why not to use mocking/stubing approach to get rid of dependency?

Answer 3

If you use dependency injection you can ensure that your class's external calls are directed towards a stub, rather than an external library.

Answer 4

Unfortunately you can't test everything in an application when unit testing. This is why when testing you aim to have your tests do a 90% code coverage over the project.

Do this test as part of your integration testing or even part of your end-to-end testing because the action of testing this will be rather expensive

Answer 5

It's not necessary to stub out your dependencies on the whole framework - just those parts that hamper your ability to unit test your code. So I think you should feel free to stub out System.IO.File operations, along with database calls. I also find it very useful to have an IClock interface to tell the time instead of calling DateTime.UtcNow - this enables us to control the passage of time in unit tests.

Answer 6

Indeed, at some point you'll have to implement interfaces like IFileSystem, thereby glueing your application to the .NET framework. This "glue-code-in-between" cannot be unit tested. This is not much of a problem as long as the glue code is very simple.

If the glue code is not as simple as you'd like, you can still do automated integration tests, e.g. run your assembled application in some sort of self-test mode.

The reason unit tests are more popular is that they are simpler to create and less fragile. You don't need to prepare a database, web server or a file system in order to run a unit test. You don't need to worry about the unit tests for class A breaking if you change class B. Integration tests can test more things but they don't have these advantages.

Answer 7

You only need to test your class, not the framework below.

For this you need a testdriver that sets up a testcase, in your example create a file to resolve, instantiate your class and let it do its work.

A good testdriver cleans up the test scenario as well and leaves the system in the same state as before the test.

Answer 8

If you can't test a class, but you can mock it using an Isolation Framework (Rhino Mocks, Typemock, Moq), you can fake out the details of that "untestable" class.

Answer 9

This is a fine example of difference between unit testing and integration testing. For unit testing your PathResolver, you need to pass a mock object created by hand or using a mock framework (such as my favorite Moq). Using a mock object, you'll able to test if Combine method was called.

The unit test will look something like this (using Moq):

[Test]
public void ShouldCombinePath()
{
    IFileSystem fs = new Mock<IFileSystem>();

    PathResolver resolver = new PathResolver(fs.Object);

    resolver.Resolve("Test.filename");

    fs.Verify(fs => fs.Combine());
}

Unit test are supposed to be executed fast without external dependencies. They should be called on every compile.

But you're right, you still need to test the concrete class. This is what we call integration testing. I suggest you create a separate project called "MyProject.IntegrationTest" and test SystemFileSystem directly using test files included in the project.

The integration test should look like this

[Test]
public void ShouldCombinePath()
{
    DotNetFileSystem fileSystem = new DotNetFileSystem();

    string resultPath = fileSystem.Combine("Test.filename");

    Assert.That(resultPath, Text.Contains("@D:\MyExample\Test.filename"));
}

Integration tests are usually called when creating a build of the software on a new commit, using a continuous integration software. They can be slow because they use external dependencies.