What makes a good test procedure for functional requirements?

Question

I'm the lead developer on a new project and have the opportunity to work with the system engineers to create our template for testing functional requirements. I was wondering if anyone had input on what makes a good test procedure template or had an example of a great template.

Thanks!

Answer 1

This isn't a very easy one to answer. It depends on a few things:

1) The definition/interpretation of what is a functional test case

2) The role of the support staff in the acceptance tests

3) The longevity of the tests

This is purely opinion based on my own experiences.

(inserts two cents into vending machine)

1) What is a functional test case? - You and the systems engineer need to align on this one. You may find (as I did) that the system engineer will tackle things at a higher (less granular) level than you. For example, assuming that a specific requirement is for the creation of a web service, the engineer would need to know:

• is the interface behave correctly?
• Are the input parameters in a test case meant to induce a success/failure?
• On failure, are the appropriate errors/error codes returned? Note that depending on their time, an engineer may only stick to major/important failure conditions (or negative responses) that affect the product/service as a whole (for example a "host not found/timeout error" should be in the interface but does not necessarily need to be tested, but a use-case related failure such as "client has insufficient funds" is important to the engineer.
• is the transaction status recorded correctly?

Again, you and the systems engineer should be clear on what is a functional test case and what is not. Usually the functional tests are derived directly from the functional spec provided to you. For certain products, retry on timeout falls under non-functional, but you may have an engineer who wants his web service to retry 17 times on a timeout before giving up - and if he specifies this - then you include it.

2) How are these tests carried out and who signs them off? Depending on this, you may need to streamline or flesh out the functional tests.

If yourself and the systems engineer will lock yourselves up into a cosy room for half a day going through each test case, then keep it streamlined: the two of you should be pretty familiar with the requirements and the engineer would have reviewed the document and have provided comment already. On the other hand, you may have the support engineers running the tests with you instead of the engineer (that's how we run it...the systems engineer reviews the test cases, stays for a bit at the beginning and leaves when he gets bored). Where was I? Right, so in this case, your document may have to go into a bit more hand-holding where you describe the scenario that is being tested. This leads me to the last point in my long winded chat...

3) Longevity of the document

So often as is the case on my side, once a set of functional tests are over and done with, they are promptly forgotten. However, these tests validate your system and your product and the support engineers should be in the position to run them whenever they'd like to :

• resolve issues ("was this sort of case even tested before go-live?")
• resolve issues again ("geez did these guys even test this particular scenario?")
• validate system/product integrity after a major change
• learn about the as-is functionality of a product or service (so many times people forget how the product is supposed to behave, and support staff hate reading requirements specs especially the requirements specs that are out of date and the current behaviour of the system differs from what was originally specced)

(deep breath)

So now you need to make sure you cover the following:

• Test set up part 1: what are the requirements to run the test? What tools do i need? network connectivity?
• Test set up part 2: what test-data am i going to use? where is it if I need it or how do i generate it?
• Overview of the functional requirements/tests to at least impart what the expected behaviour is.
• Overview of the major system components that will be tested
• An idea of the limitations of the tests - certain functional tests may only be simulated or could not be tested against a live end system etc etc - you need to decribe the limitation and show the reader how you're gonna fake it.

Also, the systems engineer will expect you to have already completed your granular tests like component tests, integration tests, etc etc as well. Depending on how awesome he is, the engineer may ask for the documentation of these component tests and run a few himself.

Hope that helps somewhat - having a template provides consistent presentation and helps you ensure that all the important content is covered - but I think the focus should be on pinning the purpose and fulfilling that purpose.

Hope I made some cents :)