Running MSTest Unit tests As part of TeamBuild

Question

I've been looking for a definitive web resource to help with this, but no luck so far. So :how do I run unit tests as part of a TFS build? I have a number of questions for this, so to make it easier I've numbered them in this post. Sorry for this post's length.

Visual Studio Team System 2008 is installed on my build machine.

I have a TFSBuild.proj file that Ive set to <RunTest>true</RunTest>. To go with this, I've added

<ItemGroup>  <TestContainer Include="$(OutDir)MyUnitTests.dll" />  </ItemGroup> 

'MyUnitTests' is the unit testing project in the solution.

The solution builds correctly, but the Run unit test step is failing with the error: "(CoreTestConfiguration target) -> Unable to load the test container 'C : \MyBuild\Binaries\Debug\MyUnitTests.dll' or one of its dependencies. warning details: "

No actual warnng details follow. I notice that the file path seems to include incorrect spaces - 'C : \MyBuild\Binaries\Debug\' in the message. The actual path is 'C:\MyBuild\etc'.

(1) Could it be that OutDir is somehow set to the wrong path? How do I determine what the OutDir target is set to on my build machine?

To make sure this is not just a file path being incorrectly set, I tried just hard-coding the path in the TestContainer: <ItemGroup> <TestContainer Include="'C:\MyBuild\Binaries\Debug\MyUnitTests.dll" /> </ItemGroup>

The error message was identical.

Other things I've tried include trying to run tests off the vsmdi file <MetaDataFile Include="$(SolutionRoot)\MySolution\MySolution.vsmdi"> </MetaDataFile>

This time there was a different error message: (CoreTestConfiguration target) -> MSBUILD : warning MSB6006: "MSTest.exe" exited with code 1.

(2) What's happening here? (3) And why am I not seeing any detail in the error messages for the unit tests/ is it because I don't have Studio 2008 Team System: Developer (or Tester) edition installed?

I'd also like to ask about specifying deployment items for running tests in a build. When run from Visual Studio, my unit tests have deployment items specified in localtestrun.testrunconfig.

(4) How do I make sure these are deployed when the unit tests run as part of the build?

(5) How can I capture the output of MSTest into a file when the build runs it? That is, have MSTest run with the command-line switch "/resultsfile:myfile.trx"?

(6) Is there any recommended web resource to learn about integrating unit tests into a build?

Update (1 day later)

I've managed to get unit tests to run as part of the build, but with problems. Here's what I've found so far

1. I was unable to get it to run using the syntax that is supposed to just run all unit tests in a project. Instead, I had to create a test list and run by referring to a vsmdi file. That is, this didn't work - no unit tests ran.

<TestContainer Include="$(OutDir)\MyProject1.Tests.dll" />

This did.

<ItemGroup> <MetaDataFile Include="$(SolutionRoot)\$(ReleaseName)\$(BranchName)\TheApplication\TheApplication.vsmdi"> <TestList>TestsForTheBuild</TestList> <TestRunConfig>$(SolutionRoot)\$(ReleaseName)\$(BranchName)\TheApplication\LocalTestRun.testrunconfig</TestRunConfig> </MetaDataFile> </ItemGroup>"

1. There were significant differences to running tests in the Visual Studio IDE and using MSTest to execute them. The problem was with project references - MSTests wouldn't deploy any references deemed by the unit test project to be 'Unused References', while the Visual Studio IDE would.

A lot of the code I'm testing instantiates objects dynamically using the Unity Application block. In order for them to be instantiated, the dll where they're defined needs to be included in the project as a reference. Because the objects are being instantiated via Reflection only at run time, as far as the compiler can see they're unused references.

When Visual Studio runs the tests, all references, whether unused or not, are compiled and copied to the Out directory that tests run from. When MSTest.exe runs the tests, it doesn't deploy any "unused" reference. It took me a couple of hours to discover dlls that were project references in the unit test project simply weren't in the Out directory when MSTest was running the tests. . It was because of this "unused reference" issue.

To solve this: include the 'unused references' dlls as deployment items in the testrunconfig file. This will force them to be copied.

1. The next problem I've found is that some unit tests fail when there are multiple solution files being compiled in the same build. I found two "flavours" of this problem. a. Unit tests weren't running at all unless the solution the related to was the last solution file in the list of solutions to compile. For example, if this was my "SolutionToBuild" ItemGroup, and I had unit tests relating to SolutionTwo, they just didn't run. <SolutionToBuild Include="$(SolutionRoot)\$(ReleaseName)\$(BranchName)\SolutionOne.sln" /> <SolutionToBuild Include="$(SolutionRoot)\$(ReleaseName)\$(BranchName)\SolutionTwo.sln" /> <SolutionToBuild Include="$(SolutionRoot)\$(ReleaseName)\$(BranchName)\SolutionThree.sln" />

To get them to start running, I had to move SolutionTwo to the botom of the list.

b. With three solutions in the build, my unit tests run, but many tests fail. If I remove the other two solutions from the build, all unit tests pass.This is despite the unit tests having nothing to do with the other two solutions.

I've figured out the answers to my questions 2 & 3 & 4 above. 2. 2. You don't see any detail in the build's output for unit tests running. You only see whether the run passed or failed. 3. To see individual test results, you need to see the TestResults directory and .trx file that MSTest generates when the build runs it. 4. Anything marked as a deployment item in your testrunconfig file will be deployed: I was confused when I asked the question by the unused references problem in MSTest.

Answer 1

I find a three step elimination is useful:

1. Do the tests run in Visual Studio dev environment?

2. Do the tests run in msbuild locally?

3. Do the tests run in msbuild on the build server?

I use the build.cmd script below to run msbuild on my local machine in an environment that very closely matches what will be used by the TFS build server.

build.cmd script

@setlocal

SET MSBUILDVER=v4.0.30319 
SET MSBUILDEXE=%FrameworkDir%%MSBUILDVER%\MSBuild.exe
SET builduri=local

%MSBUILDEXE% BuildDefinitions\TFSBuild.proj %*

NOTE: This script must be run from a Visual Studio 2010 command prompt window:

Start -> All Programs -> Microsoft Visual Studio 2010 -> Visual Studio Tools -> Visual Studio Command Prompt (2010)

In general, setting the true in TFSBuild.proj should cause unit tests to run automatically in any solutions listed in the build, but it sounds like that is not working for you?

What happens if you pass command line option /p:RunTest=true to msbuild from command shell?

You should be able to add custom TFS build command line options to the server build through the TFSBuild.rsp file if necessary, but should not really be necessary for RunTest.

TFS build server should automatically create a .trx results file -- check the build results through Build Explorer in Visual Studio -- the View Test Results will open the .trx file in VS.

Not sure about solutions to your other specific problems, but i would try running the build in a command window locally to get a clearer view of what is happening during the build. Error messages will show up as red text, warning messages as yellow text, and lots of other intermediate build step info as grey text.

There are lots of good blog posts around about TFS build subjects -- Aaron Hallberg's blog (blogs.msdn.com/b/aaronhallberg) would be a good place to start. Also, MSDN has lots of good info on TFS / MS Build - for example: Running Builds in Team Foundation Build

Answer 2

The update to the original question was the answer I finally went with.