I am currently working on a beta strategy for my mISV. I know Joel wrote that you need about 100 beta testers for a one person shop, and 100 per employee in the general sense. Is that number reasonably accurate in your experience?
That number seems high to me based on previous experience. In our last beta at my day job, we had about 75 beta testers and just managing the communication with them was nearly a full-time job.
Managing the communication is better done through something like UserVoice, or a forum.
The only time I would say use less testers is if you have a VERY specific target audience and/or your expected install base is small.
There is no exact number of testers that you need, but a more the better! and it better to have testers that are more close to your user/customer persona, that help you improve the quality and usability of your application.
How my trees are there in the Forrest?
To give exact accurate numbers which are accurate for all applications, developers, concepts is not possible. It depends...
Muhammed once was asked how long time it would take to walk from one city to another. He replied to the guy asking him; "walk"!
Then when the guy asking the question walked, Muhammed shouted after him; three days...
When confronted with why he couldn't answer immediately Muhammed said I needed to see how fast you are walking before I know how much time you'll spend to get there...
This is one of those situations....
What do you build, how experienced are you, how many lines of code are there, etc, etc, etc...
In general a more experienced developer, building something he knows very well himself, in a language he's an expert in would need few beta testers. While one using a new language, building something he doesn't understand and with only a year of development experience would need WAY more testers...
You need enough to drive out problems. Beware of the common numerology fetish of assigning exact numbers to things. The number isn't 123; it's enough.
Recently, I emailed a link to some folks. A little over 100. Of those, one -- exactly one -- reporting that the link was screwed up in their email.
I have no idea what they did wrong, but I'm guessing the copied the link out of the email and pasted it into a browser. I'm guessing the other 99% simply clicked it with no problems at all.
Here's the numerology fetish: "We need to have an error rate of under 1%. If we have 100 people and no one has an error, then the error rate is under 1%." This doesn't really make a lot of sense (since it tests people not software) but it's how weird numbers show up.
A "representative sample" might be what you want. People with appropriate backgrounds and experience. A few of each "class" or "type" of user. At this point, you should check with a medical researcher on experiment design for groups of people.
If you want to move up to "statistically significant" you'd need a solid two dozen of each "class" or "type". Depends on what you're measuring and how wide the standard deviation of that measurement is.
If you're not measuring anything, then the exact number of beta-test users is little more than numerology.
Your sense of "how many can we manage" is the right way to judge the size of the beta population.
No specific number of beta tester! one thing is it is never enough how much ever you test... so ...... and It depends on the ones u hire. better hire people with the right domain knowledge.