Failing NUnit test when running through CC.NET

Question

The solution for this error has escaped me for several days now, and it is time to come here for help. The short version is, I have a unit test that fails on the build server but no other environment.

The method I'm testing is an extension method for ILog in log4net. The purpose of this extension method is to make a debug log of the current method, when called and I use it for debugging. The code to do this is pretty straight forward.

public static void MethodHead(this ILog log, params object[] parameters)
{
    /* Assert */
    log.AssertNonNull();

    /* Since this is an expensive operation, don't do it if Debug is not enabled */
    if (log.IsDebugEnabled)
    {
        StackTrace stackTrace = new StackTrace();

        /* Get calling method */
        MethodBase method = stackTrace.GetFrame(1).GetMethod();

        string logMessage = string.Format("{0}.{1}({2})", method.DeclaringType.Name, method.Name, parameters.ToDelimitedString(", "));
        log.Debug(logMessage);
    }
}

In this method I check that debug mode is enabled, because I don't want to do StackTrace if nothing is supposed to get logged (because of performance issues). When I test this method I will use Rhino Mocks to mock the ILog interface and let IsDebugEnabled return true.

Please consider following NUnit test method.

[Test(Description = "Verify that MethodHead extension method will log with calling class.method(arguments)")]
public void MethodHeadShouldLogCurrentMethodNameWithArguments()
{
    /* Setup */
    MockRepository mocks = new MockRepository();
    ILog log = mocks.CreateMock<ILog>();
    string[] arguments = new string[] { "CAT", "IN", "A", "HAT" };

    string logMessage = string.Format("{0}.{1}({2})",
        "MethodHeadTest", "CallingMethod", arguments.ToDelimitedString(", "));

    With.Mocks(mocks).Expecting(delegate
    {
        /* Record */
        Expect.Call(log.IsDebugEnabled).Return(true);
        Expect.Call(delegate { log.Debug(logMessage); });
    })
    .Verify(delegate
    {
        /* Verify */
        CallingMethod(log, arguments);
    });
}

private void CallingMethod(ILog log, params object[] arguments)
{
    log.MethodHead(arguments);
}

This executes well in my development environment, Visual Studio 2008 with TestDriven.NET. It does execute well if I run the test through nunit-console.exe or nunit-gui. It even runs well if I use my NAnt script to execute the test.

However, my build server fails this test when it runs through NAnt which is executed from CruiseControl.NET. When I run it manually with nunit-console.exe on the build server it succeeds.

The error and stack trace are the following.

Rhino.Mocks.Exceptions.ExpectationViolationException : ILog.Debug("**<>c__DisplayClass8.<MethodHeadShouldLogCurrentMethodNameWithArguments>b__5**(CAT, IN, A, HAT)"); Expected #0, Actual #1. 
ILog.Debug("MethodHeadTest.CallingMethod(CAT, IN, A, HAT)"); Expected #1, Actual #0.

at Rhino.Mocks.MethodRecorders.UnorderedMethodRecorder.DoGetRecordedExpectation(IInvocation invocation, Object proxy, MethodInfo method, Object[] args)
at Rhino.Mocks.MethodRecorders.MethodRecorderBase.GetRecordedExpectation(IInvocation invocation, Object proxy, MethodInfo method, Object[] args)
at Rhino.Mocks.Impl.ReplayMockState.DoMethodCall(IInvocation invocation, MethodInfo method, Object[] args)
at Rhino.Mocks.Impl.ReplayMockState.MethodCall(IInvocation invocation, MethodInfo method, Object[] args)
at Rhino.Mocks.MockRepository.MethodCall(IInvocation invocation, Object proxy, MethodInfo method, Object[] args)
at Rhino.Mocks.Impl.RhinoInterceptor.Intercept(IInvocation invocation)
at Castle.DynamicProxy.AbstractInvocation.Proceed()
at ILogProxy86e676a4761d4509b43a354c1aba33ed.Debug(Object message)
at Vanilla.Extensions.LogExtensions.MethodHead(ILog log, Object[] parameters) in d:\Build\Mint\WorkingDirectory\Source\Main\Vanilla\Extensions\LogExtensions.cs:line 42
at Vanilla.UnitTests.Extensions.LogExtensions.MethodHeadTest.<>c__DisplayClass8.<MethodHeadShouldLogCurrentMethodNameWithArguments>b__5() in d:\Build\Mint\WorkingDirectory\Source\Test\Vanilla.UnitTests\Extensions\LogExtensions\MethodHeadTest.cs:line 99
at Rhino.Mocks.With.FluentMocker.Verify(Proc methodCallsToBeVerified)
at Vanilla.UnitTests.Extensions.LogExtensions.MethodHeadTest.MethodHeadShouldLogCurrentMethodNameWithArguments() in d:\Build\Mint\WorkingDirectory\Source\Test\Vanilla.UnitTests\Extensions\LogExtensions\MethodHeadTest.cs:line 90

So the problem is that the build server thinks that this method has another (dynamic?) name. Or rather it is Rhino Mocks that makes this assumption?

I don't get anywhere with this error since I can't recreate it on my development machine. I'm happy for all input I can get.

Thank you!

Mikael Lundin

Answer 1

Looks like CallingMethod was optimized away on the build server. When you repeated the test manually, did you really use exactly the same assembly?

Answer 2

That CallingMethod might get optimized away, is something that I did not think of. Let me run a few more tests on this.

First I invoke nant manually on the build server.

"C:\Program Files\nant-0.86-beta1\bin\nant.exe" test -D:nunit.exe.path="C:\\Program Files\\NUnit 2.4.8\bin\\" -D:Artifact.Output.Path="D:\\Build\\Mint\\Artifacts\\" -D:msbuild.logger="C:\\Program Files\\CruiseControl.NET\\server\\ThoughtWorks.CruiseControl.MSBuild.dll" -D:fxcop.exe.path="C:\\Program Files\\Microsoft FxCop 1.36\\"

This works fine and the test does not fail! I go to the produced binary and execute NUnit manually on it.

D:\Build\Mint\WorkingDirectory\Source\Test\Vanilla.UnitTests\bin\Debug>"C:\Program Files\NUnit 2.4.8\bin\nunit-console.exe" Vanilla.UnitTests.dll
NUnit version 2.4.8
Copyright (C) 2002-2007 Charlie Poole.
Copyright (C) 2002-2004 James W. Newkirk, Michael C. Two, Alexei A. Vorontsov.
Copyright (C) 2000-2002 Philip Craig.
All Rights Reserved.

Runtime Environment -
   OS Version: Microsoft Windows NT 5.2.3790 Service Pack 2
  CLR Version: 2.0.50727.3082 ( Net 2.0.50727.3082 )

..............................................................
Tests run: 62, Failures: 0, Not run: 0, Time: 7.891 seconds

And it works as it should. But when I force a build through CC.NET the test fails like I've shown above. Then if I pick the binaries produced by the build server and run NUnit on those I'm back at success.

So, the binaries does't change, but the test succeeds/fails depending on if NAnt is run through command line or CC.NET.

This is the cc.net task I use to execute my NAnt build script.

<nant>
    <executable>C:\Program Files\nant-0.86-beta1\bin\nant.exe</executable>
    <buildArgs>-D:nunit.exe.path="C:\\Program Files\\NUnit 2.4.8\bin\\" -D:Artifact.Output.Path="D:\\Build\\Mint\\Artifacts\\" -D:msbuild.logger="C:\\Program Files\\CruiseControl.NET\\server\\ThoughtWorks.CruiseControl.MSBuild.dll" -D:fxcop.exe.path="C:\\Program Files\\Microsoft FxCop 1.36\\"</buildArgs>
    <nologo>true</nologo>
    <buildFile>Mint.build</buildFile>
    <targetList>
     <target>clean</target>
     <target>build</target>
     <target>test</target>
     <target>staticAnalysis</target>
    </targetList>
    <buildTimeoutSeconds>1200</buildTimeoutSeconds>
</nant>

The task to execute NUnit in my build script is a bit messier.

<!-- Run all tests -->
<target name="test" description="Run NUnit tests" depends="build">
  <property name="Failed.Test.Count" value="0"/>

  <!-- Test Vanilla -->
  <property name="Test.Name" value="Vanilla.UnitTests" />
  <call target="runCurrentTest" />

  <fail if="${int::parse(Failed.Test.Count)>0}" message="Failures reported in unit tests" />
</target>

<!-- Utility method to run tests -->
<target name="runCurrentTest">   
  <exec program="${nunit.exe.path}nunit-console.exe"
      failonerror="false"
      resultproperty="Test.Result"
      verbose="true">
      <arg value="${Test.Path + Test.Name + '\bin\Debug\' + Test.Name}.dll" />
      <arg value="/xml:${Artifact.Output.Path + Test.Name}-nunit-results.xml" />
      <arg value="/nologo" />
  </exec>
  <property name="Failed.Test.Count" value="${int::parse(Test.Result) + int::parse(Failed.Test.Count)}"/>
</target>

Can I explicitly put optimization flag on the msbuild exec call, to determine that it is not a problem like that? I do point the compilation directly to the csproj file. Shouldn't msbuild pick the optimize configuration from there?

Thank you for all your input! Mikael Lundin

Answer 3

I solved it.

Removed the CallingMethod from my code and let the tests call the SUT directly. It makes the test code look a bit uglier, but it works.

Still don't know why CallingMethod changed its name when running through CC.NET. I guess that will be for someone else to figure out.