MSTest + MSBuild + Many test projects

Question

I am responsible for maintaining the msbuild scripts for a large project. The solution contains about 90 projects each having their own test project.

As part of the build process all test projects are agregated and mstest is called once:

mstest /textcontainer:project1 /testcontainer:project2 ...

This is no longer a viable solution as the constructed command is now about 12,000 characters long which exceeds the max length for a single command.

We have several options:

1. creating a custom task to split the list of projects in a logical location and calling mstest twice.
2. calling mstest once for each test project.

Are there any advantages/disadavantages to either option? Or possibly alternative solutions? NOTE: I do not have the ability to make changes to the project architecture, only the build scripts.

Answer 1

Caveat: I'm not familiar with MSTest, but with build automation in general.

Reasons to run all tests from a single test runner

• Easier to pass/fail a build in one spot.
• Avoids the overhead of spawning the test runner multiple times.
• Reporting tools may want all test results in a single location (file) for a single report.

Reasons to run each test from its own test runner

• Simpler configuration for a test project.
• Easier/faster to re-run a given set of tests.
• Test runners can be distributed across multiple servers to speed testing.

As long as you can aggregate all of the test results into a single report OR that doesn't matter to your team, I'd recommend splitting things out as finely as possible (down to one test runner per test project if that's feasible).

If you've purchased the whole MS Team Foundation Server and MS Test Manager tool set, I think it will support a broad range of testing options; other frameworks such as Gallio may suit your needs without quite so much cost or overhead.

References

• MSTest.exe docs
• TestToolsTask docs
• MSDN: Visual Studio Application Lifecycle Management: Testing the application
• Gallio Test Automation Framework