Automatically Unit Test Example Code

Question

My team is responsible for the development of an API for a large system that we also write. We need to provide example code so that other developers using our API can learn how to use it. We have been documenting the code using the xml document comments. eg.

/// <summary>Summary here</summary>
/// <example>Here is an example  <code>example code here</code> </example>
public void SomeFunction()

We then use Sandcastle and build the help files we need (chm and an online website).

It is quite embarrassing when the example code doesnt work, and this is usually because some functionality has changed or a simple error.

Has anyone ever done something like this, but also configured unit tests to run on the example code so that they are known to work during the build?

Answer 1

Simple solution: Make a small application in which you include all the sample code headers and then call their respective entry points

#include "samples/sampleA.h"

void main()
{
  SomeFunction();
}

then after you make a build run these little apps you need to be sure they ran ok. But can you verify that the code ran ok without having someone have a slumber party with the NightlyBuild server?

Better Solution: Log the output and have someone look at it in the morning.

Even Better Solution: Log the output and grep it or something so no one has to look at it unless its broken.

Best Solution: Find a suitable test framework, hopefully something with all the bells and whistles you can get so it can email people if its broken or something like that. In our case we avoid the bells and whistles instead we connected a USB Police Siren that goes off when something breaks It's quite exciting!

Answer 2

I have not done this myself but I've seen this mentioned in the Pragmatic Programmers books. If I'm not mistaken the book "Pragmatic Unit Testing in C# with Nunit" mentions that they did this for the book. It could be that they mentioned in one of their podcasts though.

They mentioned they had a continuous build server set up for their books. If I'm not mistaken they used latex or some other text based markup to write their books in and they had build steps for formatting the markup and building and unit testing code in the book.

Answer 3

I would suggest using a special bit of markup in your XML which says, "Grab the code sample from this place". It would refer to a normal C# file which can be run with unit tests. To take your example, you might have:

/// <summary>Summary here</summary>
/// <example>Here is an example
/// <code>!!sourcefile:SomeClassTest.cs#SomeFunction!!</code></example>
public void SomeFunction()

Your unit tests run as normal, and then insert a build step between "create XML" and "run Sandcastle" you'd replace each "file token" with the appropriate contents. There may even be hooks you could put into Sandcastle to do this at doc generation time - I don't know enough about Sandcastle to know for sure.

It's ugly to invent your own markup of course, but it should work.

Of course, this assumes that the code examples are easily unit testable - some may not be (if they're dealing with resources etc). At least you'd know it compiles though :)

Answer 4

Yes, sandcastle supports this and it's great to maintain the correctness of examples. You can point to a code region like this:

   /// <summary>
   /// Gizmo which can act as client or server.
   /// </summary>
   /// <example>
   /// The following example shows how to use the gizmo as a client:
   /// <code lang="cs"
   ///    source="..\gizmo.unittests\TestGizmo.cs"
   ///    region="GizmoClientSample"/>
   /// </example>
   public class Gizmo

You can then use some test code in TestGizmo.cs as an example by enclosing it in a region:

[Test]
public GizmoCanActAsClient()
{
   #region GizmoClientSample
   Gizmo gizmo = new Gizmo();
   gizmo.ActAsClient();
   #endregion
}

Caveat: If you move or rename the test file, you will only get an error about this when you try to regenerate the documentation with sandcastle.