Does anyone have any rules of thumb for esimtation of UAT effort - like what's a rule of thumb for number of scripts as compared to the number of business requirements?

Question

I'm attempting to estimate the number or testers required to test out a project. One method is to determine the number of scripts that will be required and was wondering if there was a rule of thumb for number of scripts as compare to number of requirements. I'm estimating 2 - 3.

• 1 for a sunny day type test
• 1 for a negative test
• 1 for at least combining 1 requirement test with at least one other.

but that's just my initial guess. If there are some best practices, I'm all ears. Again, this is not for unit testing or systems testing, this is for user acceptance testing.

Answer 1

The best estimate you will get comes from the testers doing the testing. Outside of that type of estimate from the testers, you may be able to come up with some sort of percentage of testing time compared to development time.

Say that you had a 100 hour development task. You spend 20 hours on design, 80 hours on build. You might be able to come to the conclusion that it will take 15 hours to test, or 15% of development time. You could then apply 15% to your overall development estimate for UAT testing knowing that some will take longer, some less.

Answer 2

Hey Shinyfish, I understand the impulse to want a formula...and I'll promise you that any general formula will be provably wrong outside of a narrow context. Consider someone who tells you that every requirement should have N tests associated with it. Now consider a few sample requirements, e.g.

1. The username field requires a minimum of 6 alphanumeric characters, vs.
2. The dosage calculator will correctly calculate the patient's dose of Danger-o-medicine, based on their age, gender, body weight.

Both are possible requirements. The first is relatively straightforward and fairly low stakes. The second has many potential points of failure, and if it succeeds in many cases but fails in a few seemingly random ones, it'll kill someone. Anyone who tells you to count your requirements and then multiply by something is deluded or selling snake oil.

Similarly, saying that UAT will take 1/Nth the coding time /may/ be a useful heuristic within some business context, but the value for N will vary wildly between, say, a startup making blogging software and developing the next version of Photoshop. For that matter, what /you/ mean by UAT (and what your unit and system testing does(n't) cover) likely varies dramatically from what the folks advising you mean by the same terms.

Here's rule of thumb that I might use to estimate how much time testing'll take:

First, to the extent that it's possible, consider similar projects within your organization.

• How many person/days of testing did they get?
• Were stakeholders happy with how thoroughly the product was tested?
• How do you expect this new project to be similar/different from previous ones?
• How experienced/skilled are the testers you'll be getting, compared to previous projects?
• How well will they understand this project at the outset, compared to previous projects?

Of course, sometimes you don't have relevant previous projects to compare to. If you don't...know that your estimate will have a much larger margin of error. I can't speak for you, but 98.% of the developers (testers, coders, etc.) that I've worked with chronically underestimate. If that's true for you, try to compensate accordingly. Perhaps most importantly, try to understand how accurate your estimate is (or isn't) and then set stakeholders' expectations accordingly. Providing an illusion of certainty rarely helps anyone.

Best of luck!

Answer 3

I would recommend coming at it from a couple different angles and making your decision after you consider the following:

1) Your back of the envelope calculation... 2.5 test cases per requirement (but Jeff Fry's point is dead on, sometimes more will be required, sometimes less)

2) Quickly calculate a 1/Nth the time answer... What percent of overall development time and/or overall testing time did we use last time for a project of this general type? Was it enough to do the job well?

3) Spend an hour inputting the parameters and values into a test design tool like Hexawise and create a simple 2-way (or pairwise) set of test conditions. Doing so will generally give you a minimum number of tests, beyond which you wouldn't typically want to cut. The additional benefit of using a test design tool is that you will not only confirm that Stated Requirement #1: "the site looks OK when using the Google Chrome browser" and Stated Requirement #2: "users are able to change the credit card they are using to pay at the end of the transaction" will be tested, but also the /Unstated/ Requirement (that no one thought to include) "Make sure that a user using Google Chrome will be able to change their credit card" gets tested as well. Expedia apparently did not follow this approach, but I digress... (Relevant side note: If you haven't tried pair-wise test design methods or a tool like Hexawise to help with semi-automated test case generation before, you should expect to see a significant increase in your testing efficiency when you begin using it; there will be fewer tests required to achieve all the possible pairs than you would think would be possible. A case in point: only 35 tests are required to achieve this type of full pairwise coverage as compared to the 72 billion tests that would be required for comprehensive testing in a test of the "Get Directions" functionality of Google Maps). Most of your requirements will fit easily into the test design tool. Some won't.

4) Take an average of the three estimates and add 20% to it if similar projects have been significantly underestimated.

Justin

Disclosure: I'm the founder of Hexawise which offers a free version of our test design tool at www.hexawise.com/users/new