QA to dev ratio

Question

We have a pretty complex Java system which includes a few back-end tiers including a database and a proprietary Swing front-end. There are back-end APIs to which external parties can attach that mimic our front-end. There are approximately 5 silos within our org that share this system. In total there are around 15 developers maintaining this system.

Is there a rule of thumb on what the size our QA team should be?

Edit: To add a bit of context based on questions raised in the responses thus far:

1. We have about four major releases a year, and a bunch of minor ones in between.
2. Our platform has money changing hands, so there are calcs that mean a lot to our customers.
3. We do use a formal bug tacking system.
4. We don't utilize TDD.
5. We use tools like cruisecontrol for continuous integration.

Answer 1

It likely depends on how involved the code base is (whether there are lots of integrating modules) and how involved the user experience is (if there are lots of visual elements and input controls to test).

If it's a pretty complex system, you may want to consider 2-3 QA engineers for every 5 developers. If however, the features don't really change much or are less involved, I would think you could get away with just 1 QA for every 5 developers (maybe even for every 10 developers).

Answer 2

No, there is no rule of thumb. Every organization is different, as every organization has a completely different set of requirements, needs, projects, etc. You are only going to be able to find what works for your organization.

Answer 3

Start off with a 1 QA to 5 or 6 developers. Then start tweaking it depending on how much work you think the QAs are having to do or having nothing to do.

In truth, that's just based on my own experience because there's too many factors involved. How developed or stable is your codebase? How big is it? How comprehensive does the testing need to be? What sort of testing and analysis tools are involved? How often are milestone releases? What sort of development process do you have? How much manual testing and how much automated testing?

There's a lot more questions. So just start with an arbitrary number and go from there.

Answer 4

As a former test team lead, I would suggest having as much testing as you can. It sounds like a lot of people in your organization depend on your software. Test early and test often.

It's very important to realize that testing is everyone's responsibility. Developers need to be writing good unit tests. UI developers should be manually testing the UI.

I try to encourage test driven development, keep eyes on metrics (using a formal bug tracking system, tracking defect densities, etc), setup a continuous integration service, and design code for testability (use a framework like Spring for dependency injection, use mocks and stubs for external services, etc - I'd be happy to discuss in more detail). The cost of fixing a bug gets exponentially more expensive the later you find it, so it's best found before it gets to formal QA.

Jeff

Answer 5

A rational QA to dev ratio can only be derived relative to the complexity of an application in question.

Clearly, an uberdeveloper can probably come up with a complex application where only 3 QAs full time can efficiently test because of the complexity of the workflow; in turn, there can be 1 QA for 3 junior devs piecing together a relatively simple input-output report application.

So the number of QA personnel you need will be determined by the number and complexity of requirements you have, and not the number of software developers you use.

Answer 6

I don't know if his point of view has changed since this article 9 years ago, but Joel would recommend 1 full time QA person per 2 full time devs. I'd say, with a system that's constantly evolving, with regular updates, this ratio is pretty good.

Then again, if you're system's not changing that often, rarely updating, why would you have many devs to begin with? I've mamaged to convince myself that 1:2 is the perfect ratio in most all situations. :)

Answer 7

2 test engineers per 3 developers is a decent start. If many test cases involve stuff that can't be automated (physically plugging/unplugging stuff, ocular verification of UI rendering) add 1 tester per 2 test engineers.

Answer 8

A lot of it depends on what the system does and what the consequences of defects is? Are you writing financial software? Or social networking? The likely cost of one failing are greater than the other so the QA effort varies. Similarly if it's a product (with multiple installed user bases which might be harder to patch) you need it to be more stable than if it's internal.

You also need to ask are you expecting them to do basic system testing or volume and load testing too. It sounds like you're looking at a combination of UI testing and API testing.

Tgive you a starting point, assuming that you're talking a "normal" type of system and we're talking basic system testing, the metric I'd go with is that 33% of all effort should be test effort. In theory this gives you a 1 to 2 ratio but assume that your developers are unit testing perhaps stretch it to 1 to 3. Certainly I'm currently running with 1 to 4 and that's not enough (I'll be rectifying it as soon as the business allows me to).

But you also want to consider how mature your software development processes are and what your specifications are like. As well as having the right number of testers you need to give them the right information to test against - if they don't have that then they're not going to be able to do a job and it's arguably wasted money.

One other option to throw in there - it sounds like the product has been under development for a while. As well as a core of regular testers, you might want to bulk up for an initial major test using short term contractors. If you're only just bringing test resource in you don't want to start them with a major backlog.