NUnit: Running multiple assertions in a single test

Question

I have been asked to write a testing application that needs to test a new stored procedure on multiple rows in a database, in essence I want to do something like this:

[Test]
public void TestSelect()
{
    foreach(id in ids)
    {
        DataTable old = Database.call("old_stored_proc",id);
        DataTable new_ = Database.call("new_stored_proc",id);

        Assert.AreEqual(old.Rows[0]["column"],ne_.Rows[0]["column"]);
    }
}

When I run this test, if 1 row doesn't match the other, the entire test fails; instead I would like to count how many times the assertion was passed and how many times it has failed. Is there a way to do this with NUnit?

I realize that NUnit might be overkill and this is a simple task without it...I just wanted to learn it. ;)

Answer 1

Well you could declare a counter and then assert the value of the counter to determine pass/fail

Also, you could do the bulk of the work in the test setup, and then just create multiple tests.

I'm not clear as to why you need all the assert stmts in the same test.

Answer 2

I would count the number of rows which do not match and then would write an assertion which will compare this number with 0 and would return the number of non matching strings in the message.

you could also use Assert.Greater for this.

P.S. In principal you should try to do one assertion per unit test. That's the gist of it.

Answer 3

Seems like you are just Asserting the wrong thing. If you want to check all the values and then assert that there are no errors (or show the number of errors) then try this:

[Test]
public void TestSelect()
{
    int errors = 0;
    foreach(id in ids)
    {
        DataTable old = Database.call("old_stored_proc",id);
        DataTable new_ = Database.call("new_stored_proc",id);

        if (old.Rows[0]["column"] != new_.Rows[0]["column"])
        {
            errors++;
        }            
    }

    Assert.AreEqual(0, errors, "There were " + errors + " errors.");
}

Answer 4

1) If the id's are constant and not looked up at test run time, create a separate unit test fixture for each id. That way you will know which id's are actually failing. See here for a write up on the problems with data driven tests:
http://googletesting.blogspot.com/2008/09/tott-data-driven-traps.html

2) If you need to dynamically look up the id's making it impossible to create a fixture for each id, use akmad's suggestion with one change. Keep a list of id's where the values are not equal and add the list to the error message. It will be extremely difficult to diagnose a failing test that only states the number of errors, as you won't know what id's cause the errors.

3) I don't know how difficult it would be to do in NUnit, but in PyUnit, when we need to run tests on dynamically generated data, we dynamically create tests fixtures and attach them to the TestCase class so that we have a failed test for each piece of data that does not pass. Though I imagine this would be much more difficult without python's dynamic abilities.

Answer 5

Based on the objective you laid out, the entire test should fail if one row doesn't match another. Counting the number of times an assertion passes or fails gives you less information than a comparison of the outcome you expected with the outcome you actually got.

Answer 6

I know that the question is specifically about NUnit, but interestingly enough, Gallio/MbUnit has a feature which allows to run and catch several assertions at once.

[Test]
public void MultipleTest()
{
    Assert.Multiple(() =>
    {
       Assert.IsTrue(blabla);
       Assert.AreEqual(pik, pok);
       // etc.
    }
}

The Assert.Multiple is catching all the failing assertions and is going to report them at the end of the test.