How are you generating tests from specifications?

Question

I came across a printed article by Bertrand Meyer where he states that tests can be generated from specifications. My development team does nothing like this, but it sounds like a good technique to consider. How are you generating tests from specifications? How would you describe the success your having in discovering program faults via this method?

Answer 1

This might be a reference to RSpec, which is a really clever way of developing tests as a series of requirements. I'm still getting used to it, but it's been very handy in both defining what I need to do and then ensuring I do it.

Answer 2

@Tim Sullivan from Bertrand Meyer it can only be related to Eiffel :)

I think he's talking about ESpec. Given the name RSpec from the Ruby Folk, I think we can give them the label "heavily inspired".

Answer 3

I would say it depends on your specs. I have yet to work anywhere where the specs were good enough to create full unit tests from specifications - the level of detail just wasn't there. My managers always told us that if we specified to that level they could just ship the specs off to India and get it coded on the cheap ;)

Answer 4

There are all sorts of ways to do it, ranging from what I'd consider an 'art form' (and not necessarily good art) all the way to mathematically derived tests from formal specifications. At the end of the day, your development team needs to decided on what they can do based on the schedule they are working with. That being said, being able to test software against specs is a Good Thing.

Only your team can gauge the 'depth' of your tests, and that will probably be a function of how good your specs are. If they say something like, 'the login UI needs to provide a cancel button and a login button, and they need to work', your tests are going to be pretty general. But keep in mind - even very general tests are a Good Thing. Testing is a Good Thing. Too many developers have a bad attitude when it comes to testing, but at the end of the day, you're shipping software which should work, and to me, that means a lot.

The effectiveness your tests will having in finding program faults will depend on the detail you put into them. What is especially nice about having test procedures written to specs is that you can test each build to the same level of detail as the previous build (typically referred to as a regression test).