What is the best method for storing test cases that your xunit tests load from ?

Question

This is a "design/architecture" question and so you don't need to provide me technical details.

I can find examples on the internet all day long that show how to run a xunit unit test, log test results, and how to show if it succeeds or fails but every example on the internet is hard coded and nobody shows examples of how you store your test cases (nunit or junit) efficiently in a way that makes it possible to build hundreds or thousands of tests and load them into your test engine.

I assume this would be done using an XML file or a database table. Can any of you share what you think is the best method for storing tests that will be loaded by test code? I need to be able to write a testing framework that will ultimately have thousands of tests for a typical web workflow app, and I want to do it the right way.

Maybe describe your answer in the context of doing unit tests of the "Google Advanced Search" page, which has many options, and hundreds of variations, etc.

Answer 1

If you're getting at "data driven tests", then you should take a look at the RowTest approach and equivalents. NUnit borrowed this from MBUnit. xUnit seems to have a even more extensible solution to this via the Theory and PropertyData, where you don't have to specify the inputs in code ; you could generate the inputs from an arbitrary function, spreadsheet or SQL Server database.

I've never had the need for anything over the Rowtest method (which is also rare).. so can't warn you of any dragons. Alternatively you could even use Fit/Fitnesse to do this if you can't do the above approaches ; although not technically an acceptance test.

As for the 'testing framework', maybe YAGNI maybe you do. Get started, start small, keep it simple, grow the test design with time.

Answer 2

I agree that the important thing here is to keep it simple, and let it grow 'naturally'. Tests are as important as your other code, and should be refactored just as vigorously.

If what you're talking about is actual unit tests (as opposed to integration tests where you test the whole stack from the UI through to the database) it is very important that you keep everything necessary to run them in your solution, so that developers will always have the newest stuff when updating their source files. Also, you should aim at a no-thrills setup scenario, where new developers can just get the source files from the version control system and run the tests with no setup required.

The RowTest feature is really good for simple scenarios, but if you're dealing with complex scenarios you should look into the Test Data Builder pattern that lets you define default scenarios and then variations of these.

I like to keep my test data as close to the tests as possible, thus allowing me to really see what is going on in the tests. Therefore I don't think the question is whether you should use XML or a database, they're just different formats for representing data, but whether you really want to separate your test data from your tests?