How to report the progress when NUnit tests crashes on a CruiseControl.NET server?

Question

Nunit works quite well with CruiseControl.NET, but there is one thing that irritates me a lot. If there is a test that causes Nunit to crash, I would only get little information about the crash because the XML report of Nunit doesn't get a chance to be created and be merged into the CruiseControl report.

I need a way to report the progress even when Nunit crashes during the execution.

I have been tried to force each test to output some information to the console to resolve this problem. I have thought about using SetUp method, but I haven't found any good way to get the name of the current running test.

Answer 1

If getting the name of the current running test is what you're after you could grab it with the following:

using System.Diagnostics;

...

[Test]
public void SomeTestThatWillCrash()
{
    StackFrame sf = new StackFrame();
    Console.WriteLine("Now running method: " + sf.GetMethod().Name);
    ...

}

Answer 2

CruiseControl.net recommends that you use NUnit through your builder (i.e. NAnt/MSBuild). See here: http://confluence.public.thoughtworks.org/display/CCNET/NUnit+Task. As they describe - it will allow you to run these tests locally first - which should give you an exception that you can clear up.

That being said - are your developers running these unit tests prior to checking in code? That could ease this issue. If its an integration issue - I would suggest grabbing the latest code base and running the tests locally to see what is out of sorts.

I don't know if NUnit is able to create the results file even when it crashes. Even if it did - you could run into problems if that file is not well formed due to the crash.

Answer 3

You could use @jpoh's approach but do it in the TestSetup method which would require you do it per-fixture. If really needed, you could write a base class that all your test fixtures inherit from that implement this method.

Another solution is to use MSBuild to run NUnit and use the task in the MSBuildCommunityTasks library. This allows you to continue on error and also get the error code back from NUnit. You won't get what method caused the problem, but might help some. Here is my MSBuild target:

  <Target Name="UnitTest"
      DependsOnTargets="BuildIt">

    <NUnit Assemblies="@(TestAssemblies)"
        ToolPath="$(NUnitx86Path)"
        WorkingDirectory="%(TestAssemblies.RootDir)%(TestAssemblies.Directory)"
        OutputXmlFile="@(TestAssemblies->'%(FullPath).$(NUnitFile)')"
        Condition="'@(TestAssemblies)' != ''"
        ExcludeCategory="$(ExcludeNUnitCategories)"
        ContinueOnError="true">
      <Output TaskParameter="ExitCode" ItemName="NUnitExitCodes"/>
    </NUnit>

    <!-- Copy the test results for the CCNet build before a possible build failure (see next step) -->
    <CallTarget Targets="CopyTestResults" Condition="'@(TestAssemblies)' != ''"/>
    <Error Text="Test error(s) occured" Code="%(NUnitExitCodes.Identity)" Condition=" '%(NUnitExitCodes.Identity)' != '0' And '@(TestAssemblies)' != ''"/>

  </Target>

This probably won't fit your needs as is, but is something to try out and play with.

That said, I would agree with @rifferte that it sounds like you need to debug the problem locally and not rely on CC.NET to handle the reporting.

Answer 4

I think a better answer would be to create an NUnit Add-in that implements EventListener interface to capture the TestStarted event to output the progress to the console or a file.

The EventListener interface is documented on NUnit website: http://nunit.org/index.php?p=eventListeners&r=2.5

In addition, we can make the Dashboard report better even when NUnit crashes during its execution. We can use the following procedure to ensure that the DashBoard always shows something about the tests.

1. Run tests with the EventListener which outputs the progress to a separate file
2. After running tests, use another program to check the file
3. If the file does not contain a specific "end line", generate a special XML report based on the file and merge it into the CruiseControl log