Is defining TestMethod's in test base classes not supported by MsTest?

Question

This question concerns a general unit test technique with a potentially very useful wide range of applicable scenarios. But it is easier to understand with an example to illustrate my question better.

Let's say I want to test that all types that override Equals() does so correctly. Since Equals() is defined as virtual in System.Object, a wide range of types may change that behavior. Each type that does so, will have to have tests to make sure that the new behavior follows the implicit expectations of a caller of that method. Specifically for Equals(), if you override that method the new implementation must make sure that two equal objects also have equal hash codes, as defined by System.Object.GetHashCode().

Thus to enforce this, multiple test classes will be needed and they will all test for the same consistency of behavior across all these types.

To avoid having to re-type all the TestMethods required to test such a type I instead define a base test class that looks like below, and have those test classes all inherit the same behavior test suite:

/// <summary>
/// Test fixture base class for testing types that overrides Object.Equals()
/// </summary>
/// <typeparam name="T">The production type under test</typeparam>
public abstract class EqualsFixtureBase<T>
{
    #region Equals tests

    protected static void CompareInstances(T inst1, T inst2, bool expectedEquals)
    {
        Assert.AreEqual(expectedEquals, inst1.Equals((T)inst2));
        Assert.AreEqual(expectedEquals, inst1.Equals((object)inst2));
        if (expectedEquals)
        {
            // equal instances MUST have identical hash codes
            // this is a part of the .NET Equals contract
            Assert.AreEqual(inst1.GetHashCode(), inst2.GetHashCode());
        }
        else
        {
            if (inst2 != null)
            {
                Assert.AreNotEqual(inst1.GetHashCode(), inst2.GetHashCode());
            }
        }
    }

    /// <summary>
    /// Creates version 1 instance of the type under test, not 'Equal' to instance 2.
    /// </summary>
    /// <returns>An instance created with properties 1.</returns>
    protected abstract T CreateInstance1();

    /// <summary>
    /// Creates version 2 instance of the type under test, not 'Equal' to instance 1.
    /// </summary>
    /// <returns>An instance created with properties 2.</returns>
    protected abstract T CreateInstance2();

    /// <summary>
    /// Creates an instance equal to the version 1 instance, but not the identical
    /// same object.
    /// </summary>
    /// <returns>An instance created with properties equal to instance 1.</returns>
    protected abstract T CreateInstanceThatEqualsInstance1();

    [TestMethod]
    public void Equals_NullOrDefaultValueTypeInstance()
    {
        T instance = CreateInstance1();
        CompareInstances(instance, default(T), false);
    }

    [TestMethod]
    public void Equals_InstanceOfAnotherType()
    {
        T instance = CreateInstance1();
        Assert.IsFalse(instance.Equals(new object()));
    }

    [TestMethod]
    public void Equals_SameInstance()
    {
        T slot1 = CreateInstance1();
        CompareInstances(slot1, slot1, true);
    }

    [TestMethod]
    public void Equals_EqualInstances()
    {
        T slot1 = CreateInstance1();
        T slot2 = CreateInstanceThatEqualsInstance1();
        CompareInstances(slot1, slot2, true);
        CompareInstances(slot2, slot1, true);
    }

    [TestMethod]
    public void Equals_NonEqualInstances()
    {
        T slot1 = CreateInstance1();
        T slot2 = CreateInstance2();
        CompareInstances(slot1, slot2, false);
        CompareInstances(slot2, slot1, false);
    }

    #endregion Equals tests
}

I can then reuse these TestMethods for each type overriding Equals(). For instance, this would be the test class definition for testing that the System.String type implements Equals() correctly.

[TestClass]
public class ExampleOfAnEqualsTestFixture : EqualsFixtureBase<string>
{
    [TestMethod]
    public void Foo()
    {
        Assert.IsTrue(true);
    }

    protected override string CreateInstance1()
    {
        return "FirstString";
    }

    protected override string CreateInstance2()
    {
        return "SecondString";
    }

    protected override string CreateInstanceThatEqualsInstance1()
    {
        return "FirstString";
    }
}

This can also be extended further. For instance, for types that overload the == and != operators, a second abstract test base class can be defined (i.e. EqualsOperatorsFixtureBase<T> : EqualsFixtureBase<T>) that tests that the implementation of those operators are not only correct, but also consistent with the extended definitions of Equals() and GetHashCode().

I can do this using NUnit, but when using MsTest I get problems.

a) Visual Studio 2010 only discovers the Foo() test method, not the inherited test methods so it can't run them. It seems the Visual Studio test loader does not walk the inheritance hierarchy of the test class.

b) When I check in these types in TFS, TFS finds the abstract EqualsFixtureBase type and thinks it is a test class to be run. But since it can not be created, it can't run it and labels the tests in that type as inconclusive - which fails the test run, and thus the build (!).

Is there a way to get around this, or is this a limitation of MsTest and Visual Studio?

If so, is fixing this in the roadmap for VS/TFS ??

This would be very useful, especially when testing production types that implement an interface, or are part of an inheritance hierarcy, where certain members have semantic 'contract type' properties or invariants - if that makes sense.

Basically, not having support for this inhibits me from refactoring my test code to remove duplication.

Thanks

EDIT: I found this link to one of the MSDN blogs, it says the following

"In Whidbey, support for test class inheritance was missing. In Nunit, it is fully supported. This will be rectified in Orcas."

That was written over three years ago. Why has this not been added yet? I don't get it, there are legitimate reasons to have this and in my mind it would be a minor change. Or am I just not jumping the right hoops here?

Answer 1

Using VS 2010 I am not seeing the same behavior as you are. When I copied your 2 classes into a test project and compiled it I got the output:

UTA004: Illegal use of attribute...The TestMethodAttribute can be 
defined only inside a class marked with the TestClass attribute

So I marked EqualsFixutureBase:

[TestClass]
public abstract class EqualsFixtureBase<T>
{
...
}

Now it compiles without warning and when I select run tests for ExampleOfAnEqualsTestFixture it runs Foo and all 5 of the inherited equals tests. Also when I copy the ExampleOfAnEqualsTestFixture and use it for int and run the tests for the solution I see all 5 inherited tests running (and passing) for the example string class and the example int class.

Are you doing something in addition to your example which might be causing your problem?

Answer 2

Out of the box, it looks like unit test inheritance only works if the base test class is in the same assembly as the derived classes. For me, this usually defeats the purpose of having the base class. I, too, wonder why there isn't more posted about this on blogs, and if I might be missing something.

You might be able to work around the problem by linking the base class into every project where you want to use it. Maybe mark it as internal so the multiple copies don't interfere with each other.

There's also the TestClassExtensionAttribute that you can extend to hook into the test execution engine. I tried using it to reflect over the test classes and load the base class's tests, but a lot of the classes are undocumented, and I couldn't get it to work.