How many cycles are required to validate an automated script

Question

Hi,

I have one query. Maybe it is a silly question but still I need the answer to clear my doubts.

Testing is evaluating the product or application. We do testing to check whether there are any show stoppers or not, any issues that should not present.

We automate (script I am talking about) testcases from the present test cases. Once the test case is automated, how many cycle do we need to check the test that the script is running with no major errors and thus the script is reliable to run instead of manually executing the test cases.

Thanks in advance.

Answer 1

It's a huge field with no simple answer.

It depends on several factors, including:

• The code coverage of your tests
• How you define reliable

Answer 2

If the test script always fails when a test fails, you need to run the script only once. Running the script several times without changing the code will not give you additional safety.

You may discover that your tests depend on some external source that changes during the tests and thereby make the tests fail sometimes. Running the tests several times will not solve this issue, either. To solve it, you must make sure that the test setup really initializes all external factors in such a way that the tests always succeed. If you can't achieve this, you can't test reliably, so there is no way around this.

That said, tests can never make sure that your product is 100% correct or safe. They just make sure that your product is still as good (or better) as it was before all the changes your made since the last test. It's kind of having a watermark which tells you the least amount of quality that you can depend on. Anything above the watermark is speculation but below it (the part that your tests cover) is safe.

So by refining your tests, you can make your product better with every change. Without the automatic tests, every change has a chance to make your product worse. This means, without tests, your quality will certainly deteriorate while with tests, you can guarantee to maintain a certain amount of quality.