Parse Fitnesse RESTFul XML output into TFS Test format

Question

I'm integrating a Fitnesse Acceptance test suite into a TFS based CI process.

I can run the Fitnesse Test suite in a RESTful manner (http://fitnesse.org/FitNesse.UserGuide.RestfulTests):

http://myfitnesseserver/MyTestSuite?suite&format=xml

and get back an XML document of test results.

I'd like to transform that into a format that TFS can interpret as number of tests passed / failed.

Any pointers?

Thanks

Answer 1

The TRX file format for 2008 is fairly easy to generate, but not overly well documented. It contains a bunch of GUIDS - the best documentation for it is in this blog post:

• Dominic Hopton's Blog: Helpful Internals of TRX (and VSMDI files)

I've written some code that will take the output from JUnit and transform that into a TRX file. It actually does that in two steps - the first one amalgamates all the JUnit results files togther into a single file and generates the necessary GUIDS that the TRX file needs. It then runs an XSLT on the amalgamated XML file to convert it into the TRX file format before publishing to TFS using the MSTest.exe command line tool that ships with a Team edition of Visual Studio (Team Suite, Developer or Test edition).

You can download that code here license under the MSPL

• http://labs.teamprise.com/build/extensions.html

Answer 2

I have a similar goal at work, so I created a generic command-line Fitnesse test runner that executes a test or suite as a web request, parses the resulting XML and transforms it using the style sheet below, and finally writes the result to a file called "results.xml" in the %TestOutputDirectory% as specified by a Generic Test in Visual Studio.

The results file is loaded by Visual Studio and parsed as a summary results file that reports the number of child tests that pass or fail in a Fitnesse test or suite. Details of the output file format are documented here, but a simple example looks like this when run against Fitnesse's two minute example in the default Fitnesse wiki:

<?xml version="1.0" encoding="utf-8"?>
<SummaryResult>
  <TestName>TwoMinuteExample</TestName>
  <TestResult>Failed</TestResult>
  <ErrorMessage>6 right, 1 wrong, 0 ignores and 0 exceptions.</ErrorMessage>
  <InnerTests>
    <InnerTest>
      <TestName>TwoMinuteExample</TestName>
      <TestResult>Failed</TestResult>
      <ErrorMessage>6 right, 1 wrong, 0 ignores and 0 exceptions.</ErrorMessage>
    </InnerTest>
  </InnerTests>
</SummaryResult>

So, now it is possible to create a Visual Studio "Generic Test" in a test project for each Fitnesse test/suite you want to execute from Visual Studio or as part of a build. The generic test must specify the path to the generic test runner executable, the Fitnesse server, port and test/suite name in the wiki. It requires checking the box for the "summary results file" and entering "Results.xml" to get the detail to show up in the output of the test run or build.

I can share this code with you, or you could use the generic command line test runner and pipe the output into a small app that transforms the results using the style sheet below.

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet
    version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:msxsl="urn:schemas-microsoft-com:xslt"
    exclude-result-prefixes="msxsl"
    >
  <xsl:output method="xml" indent="yes"/>

  <xsl:variable name="GlobalRightCount" select="sum(//result/counts/right)"/>
  <xsl:variable name="GlobalIgnoresCount" select="sum(//result/counts/ignores)"/>
  <xsl:variable name="GlobalWrongCount" select="sum(//result/counts/wrong)"/>
  <xsl:variable name="GlobalExceptionsCount" select="sum(//result/counts/exceptions)"/>
  <xsl:variable name="GlobalFailureCount" select="$GlobalWrongCount + $GlobalExceptionsCount"/>

  <xsl:template match="testResults">
    <SummaryResult>
      <TestName>
        <xsl:value-of select="rootPath"/>
      </TestName>
      <xsl:choose>
        <xsl:when test="$GlobalFailureCount = 0">
          <TestResult>Passed</TestResult>
          <xsl:call-template name="GlobalErrorMessage"/>
        </xsl:when>
        <xsl:otherwise>
          <TestResult>Failed</TestResult>
          <xsl:call-template name="GlobalErrorMessage"/>
        </xsl:otherwise>
      </xsl:choose>
      <InnerTests>
        <xsl:for-each select="result">
          <InnerTest>
            <TestName>
              <xsl:value-of select="relativePageName"/>
            </TestName>
            <xsl:choose>
              <xsl:when test="sum(counts/wrong) + sum(counts/exceptions) = 0">
                <TestResult>Passed</TestResult>
                <xsl:call-template name="ResultErrorMessage"/>
              </xsl:when>
              <xsl:otherwise>
                <TestResult>Failed</TestResult>
                <xsl:call-template name="ResultErrorMessage"/>
              </xsl:otherwise>
            </xsl:choose>
          </InnerTest>
        </xsl:for-each>
      </InnerTests>
    </SummaryResult>
  </xsl:template>


  <xsl:template name="GlobalErrorMessage">
    <ErrorMessage><xsl:value-of select ="$GlobalRightCount"/> right, <xsl:value-of select ="$GlobalWrongCount"/> wrong, <xsl:value-of select ="$GlobalIgnoresCount"/> ignores and <xsl:value-of select ="$GlobalExceptionsCount"/> exceptions.</ErrorMessage>
  </xsl:template>

  <xsl:template name="ResultErrorMessage">
    <ErrorMessage><xsl:value-of select ="sum(counts/right)"/> right, <xsl:value-of select ="sum(counts/wrong)"/> wrong, <xsl:value-of select ="sum(counts/ignores)"/> ignores and <xsl:value-of select ="sum(counts/exceptions)"/> exceptions.</ErrorMessage>
  </xsl:template>
</xsl:stylesheet>

Update: Adding generic test runner code

This is definitely just a proof of concept and not necessarily a final solution. You may be able to combine this technique with Martin Woodward's answer to get the complete picture where the test list itself is dynamic. Or, you could alter the test runner to run all the tests it finds in an entire (sub)wiki. There are probably several other options here.

The following code is far from optimized, but shows the general process. You can paste this into a new console application project. Note that it requires command line parameters, and you can provide defaults for these in the project properties:

Sample command line: localhost 80 FitNesse.UserGuide.TwoMinuteExample

class Program
{
    static int Main(string[] args)
    {
        //Default to error unless proven otherwise
        int returnValue = 1; 

        try
        {
            List<string> commandLineArgs = args.ToList<string>();

            string host = args[0];
            int port = int.Parse(args[1]);
            string path = args[2];
            string testCommand = "suite";

            XmlDocument fitnesseResults = GetFitnesseResult(host, port, path, testCommand);
            XmlDocument visualStudioResults = TransformFitnesseToVisualStudioResults(fitnesseResults);
            visualStudioResults.Save("Results.xml");

            var testResultNode = visualStudioResults.DocumentElement.SelectSingleNode("TestResult");
            var value = testResultNode.InnerText;
            if (value == "Success")
            {
                returnValue = 0;
            }
        }
        catch (System.Exception ex)
        {
            Console.WriteLine(ex.ToString());
        }

        return returnValue;
    }

    private static XmlDocument GetFitnesseResult(string host, int port, string path, string testCommand)
    {
        UriBuilder uriBuilder = new UriBuilder("http", host, port, path, "?" + testCommand + "&format=xml");
        WebRequest request = HttpWebRequest.Create(uriBuilder.Uri);
        request.CachePolicy = new RequestCachePolicy(RequestCacheLevel.NoCacheNoStore);

        WebResponse response = request.GetResponse();
        Stream responseStream = response.GetResponseStream();
        StreamReader responseReader = new StreamReader(responseStream);
        string responseString = responseReader.ReadToEnd();

        XmlDocument rawResults = new XmlDocument();
        rawResults.LoadXml(responseString);

        return (rawResults);
    }

    private static XmlDocument TransformFitnesseToVisualStudioResults(XmlDocument fitnesseResults)
    {
        XslCompiledTransform transformer = new XslCompiledTransform(false);
        string codeBase = Assembly.GetEntryAssembly().CodeBase;
        string directory = Path.GetDirectoryName(codeBase);
        string xsltPath = Path.Combine(directory, "FitnesseToSummaryResult.xslt");
        transformer.Load(xsltPath);

        MemoryStream resultsStream = new MemoryStream();
        transformer.Transform(fitnesseResults, null, resultsStream);
        resultsStream.Position = 0;
        XmlDocument results = new XmlDocument();
        results.Load(resultsStream);

        return (results);
    }
}

Answer 3

@Jerry, et. al.

Have you run into this problem? I run code remarkably similar to that above, When in nUnitTest mode and the URI includes "/?test&format=xml" the nUnit test fails with and IOException, "Unable to read data from the transport connection: The connection is closed."

However the Fiddler trace that was running at the time shows the very xml I expected.

I've recreated the request headers exactly (almost) as they are sent when sent through the browser.

Finally, if I leave off the "/?test&format=xml" from the URI, I get the html I would have otherwise expected.

Intrigued? I have source code... :)

Answer 4

Hi I get an error when running FitNesse restful MyServer:8085/MySuite.Calculator?suite&format=xml

I get the following error:

An invalid character was found in text content. Error processing resource 'http://MyServer:8085/MySuite.Calc...

<content><![CDATA[<span class="meta">variable defined: unreadable_password=!"#

It fails when I have this variable defined: !define unreadable_password {!"#¤%&/(?=)(_}

Have you got similar errors with the xml format??

BR Niklas Pettersson