How to choose between different test types with SpecFlow, Cucumber or other BDD acceptance test framework?

Question

I am looking at SpecFlow examples, and it's MVC sample contains several alternatives for testing:

• Acceptance tests based on validating results generated by controllers;
• Integration tests using MvcIntegrationTestFramework;
• Automated acceptance tests using Selenium;
• Manual acceptance tests when tester is prompted to manually validate results.

I must say I am quite impressed with how well SpecFlow examples are written (and I managed to run them within minutes after download, just had to configure a database and install Selenium Remote Control server). Looking at the test alternatives I can see that most of them complement each other rather than being an alternative. I can think of the following combinations of these tests:

• Controllers are tested in TDD style rather than using SpecFlow (I believe Given/When/Then type of tests should be applied on higher, end-to-end level; they should provide good code coverage for respective components;
• MvcIntegrationTestFramework is useful when running integration tests during development sessions, these tests are also part of daily builds;
• Although Selenium-based tests are automated, they are slow and are mainly to be started during QA sessions, to quickly validate that there are no broken logic in pages and site workflow;
• Manual acceptance tests when tester is prompted to confirm result validity are mainly to verify page look and feel.

If you use SpecFlow, Cucumber or other BDD acceptance test framework in you Web development, can you please share your practices regarding choosing between different test types.

Thanks in advance.

Answer 1

Since acceptance tests are a kind of functional tests, the general goal is to test your application with them end-to-end. On the other hand, you might need to consider efficiency (how much effort is to implement the test automation), maintainability, performance and reliability of the test automation. It is also important that the test automation can easily fit into the development process, so that it supports a kind of "test first" approach (to support outside-in development).

So this is a trade off, that can be different for each situation (that's why we provided the alternatives).

I'm pretty sure, that today the most widely fitting option is to test at the controller layer. (Maybe later as UI and UI automation frameworks will evolve, this will change.)

Answer 2

It's all behaviour.

Given a particular context, when an event occurs (within a particular scope), then some outcome should happen.

The scope can be a whole application, a part of a system or a single class. Even a function behaves this way, with inputs as context and the output as outcome (you can use BDD for functional language as well!)

I tend to use Unit frameworks (NUnit, JUnit, RSpec, etc.) at a class or integration level, because the audience is technical. Sometimes I document the Given / When / Then in comments.

At a scenario level, I try to find out who actually wants to help read or write the scenarios. Even business stakeholders can read text containing a few dots and brackets, so the main reason for having a natural language framework like MSpec or JBehave is if they want to write scenarios themselves, or show them to people who will really be put off by the dots and brackets.

After that, I look at how the framework will play with the build system, and how we'll give the ability to read or write as appropriate to the interested stakeholders.

Here's an example I wrote to show the kind of thing you can do with scenarios using simple DSLs. This is just written in NUnit.

Here's an example in the same codebase showing Given, When, Then in class-level example comments.

I abstract the steps behind, then I put screens or pages behind those, then in the screens and pages I call whatever automation framework I'm using - which could be Selenium, Watir, WebRat, Microsoft UI Automation, etc.

The example I provided is itself an automation tool, so the scenarios are demonstrating the behaviour of the automation tool through demonstrating the behaviour of a fake gui, just in case that gets confusing. Hope it helps anyway!