How to get code coverage when testing Silverlight

Question

I'm looking for code coverage solution which works with silverligt. It must also work with msbuild because we run all the other unit tests automatically in the build server.

Integration with Visual Studio and/or resharper is a plus and I have nothing against easy setup or a pointer to a tutorial. I don't usually modify the build files and unfortunately I don't have time to learn msbuild at the moment.

Answer 1

Unfortunately, due to the transparent security model (i.e. all applications sandboxed) in the Silverlight runtime, this is not an easy task. Traditional .NET code coverage instrumentation modifies the underlying assembly, inserting P/Invoke calls to coverage collection tools.

As a result, tools like the Visual Studio 2008 code coverage instrumentation tool do not work with Silverlight.

To accomplish this in Silverlight, you would need to design a custom instrumentation program plus a way to collect and report that data.

On the Silverlight team we've developed a partial solution that may enable this for some people, but it is not ready for release. Look for it someday in the Silverlight Toolkit, possibly.

Sorry that I don't know an answer for you today.

Answer 2

According to NCover FAQ it can cover silverlight:

[http://docs.ncover.com/ref/3-0/frequently-asked-questions/#q05%5D%5B1%5D

So far I haven't found any information whether that actually works e.g. for silverlight class libraries. Strange that if ncover works there is no information (blog posts etc.) about it. Is everyone using the "Create standard .NET library and link files into it to get the coverage data" path when testing silverlight code?

When it comes to other solutions (even commercial) my googling hasn't produced any results.

Answer 3

The answer at this point is No (If Jeff Wilcox says no, it's no ;-) ).

I asked NCover support this question a while back (about their FAQ re Silverlight):

...I am using Silverlight 2.0 and AFAIK it can only run in the browser. So are you talking about SL3 or have I missed something?

I believe SL3 will be RTMed soon. If I upgrade, will my application be able to run out of the browser (and therefore use NCover) automatically, or will I have to change it to do this?

And this is the response I got:

The FAQ is accurate as it stands, however the wording makes it somewhat confusing. What is meant by the FAQ topic is that NCover does not support covering Silverlight code (v1 or v2 or v3) when using the //iis argument, because then it runs on its own CLR.

However, the code-behind and supporting classes of a Silverlight (v1 or v2 or v3) application are valid C#/VB, so you can write unit tests against that portion of the app and run those tests with NUnit/MSTest/Gallio/xUnit/etc and you will get coverage of that portion.

We have not yet investigated whether NCover can gather coverage on Silverlight v3 when it is run outside of the browser, but I expect that we won't be able to. Once we know for sure however, we will make a note of it in the documentation.

I took this to mean that actually Silverlight apps aren't supported - you have to "Create standard .NET library and link files into it to get the coverage data".

Answer 4

I asked this from NCover support.

My question was:

Lets say I create a silverlight class library (Project -> New Project...) from Visual Studio. In this class library I have several classes and I use the NUnit silverlight template to project which contains unit tests for my silverlight class library:

http://weblogs.asp.net/nunitaddin/archive/2008/05/01/silverlight-nunit-projects.aspx

I am able to run unit tests just fine (even in the build server, we are using msbuild and TFS) but I cannot get code coverage data for them using the standard code coverage options that are available in Visual Studio.

Can NCover show the code coverage for my silverlight class libraries?

Response I got:

As long as you're testing the class libraries with NUnit, you should be able to cover them using NCover. If you test them through a web browser using a tool like Selenium, then you won't be able to get coverage.

Now I'm wonder if anyone in the silverlight team has tried to use NCover + NUnit? Of course MS has their own unit test framework so in the "business sense" it wouldn't look so good :) But how about just testing whether this works.

Answer 5

Follow the following procedure and get Code Coverage results;

-Add a new test project to the solution
-Remove all the existing classes (Database test, ordered test etc..) so that you are only left with the unit test class (UnitTest1).
-Remove all useless comments and crud code from the test class so that you are only left with a test method (no comments, not even the TestContext)
-Add a reference to the silverlight versions of “System.dll”, “System.Windows.dll” to the test project. (usually located under “C:\Program Files\Microsoft SDKs\Silverlight\v2.0\Reference Assemblies\” (remove existing reference to system.dll if you need to first)
-Add a reference to the project under test
-You can now write tests against the object model (standard classes)

Cheers,

Ref: http://weblogs.asp.net/rosherove/archive/2008/12/27/unit-testing-in-silverlight-land-with-typemock-isolator.aspx

Answer 6

I am not able to get code coverage of black box test of a silverlight based dot net application