Here is my experience with MS Test
- We are running MS Test with around 3800 Test.
- It takes very long for the tests just to start executing, which is painful when running single tests.
- It takes around 1GB Memory to execute the tests. No, it is not due to memory leaks in our tests. Frequently we run into OutOfMemoryExceptions.
- Because it uses that much resource, we are starting to execute the tests from batch-files. So what's the whole integration good for?
- It is buggy and unstable:
- For instance, if you remove the [Ignore] Attribute from a test, it does not recognize it, because it caches information about tests somewhere. You need to refresh the testlist, which sometimes solves the problem, or restart VS.
- It randomly does not copy reference assemblies to theout directory.
- Deployment Items (additional files to be used) just don't work properly. They are ignored randomly.
- There is hidden (not visible in the test code) information in vsmdi and testrunconfig files. If you don't care about it, it might not work.
- Functionally it might be comparable to NUnit, but it is very expensive if you consider using VS tester edition.
Addition: We have some more tests now, can't even say how many. It is impossible to run them all anymore from Visual Studio, because of OutOfMemoryExceptions and other instability problems. We run the tests from scripts. It would be easy to view test results in Visual Studio, but when the solution is open, VS crashes (every time). So we need to search the failing tests using text search. There is no advantage of an integrated tool anymore.