Any benefit/disadvantages to open beta vs. closed beta for webapp

Question

This isn't a question about code, but it's programming related. We have a web app that's ready for beta testing. Has anyone noticed any difference between open beta vs. closed beta in terms of the quality or quantity of feedback the testers give or any other factors?

Answer 1

With a closed beta the people generally want to be there. They've either waited patiently in line to get an invite after signing up, or they've forgotten about it by the time it starts.

With an open beta, you get more people but they tend to be the "hey, this website is neat, I think I'll... OOH SHINY! runs off after a piece of tin foil" type.

Answer 2

With a closed beta, you can limit the number of users

This might not seem like a big deal but, consider this...

Closed Beta:

• the user base is selected by requiring them to write a proposal in how they expect to use the app
• you release a highly anticipated app to 100 users with no extra invites for the first month
• those users use the program on a regular basis and pick through most of the really common bugs that slipped through the pre-beta release QC.
• those users feel privileged to use the app so they brag to everybody about how much they love it (tons of free PR) and are less inclined to trash it because it's still a 'closed beta for a reason'
• most of the commonly occurring bugs are identified in the first month and a limited number of invites are given to the first set of beta testers to progress into step 2 of beta
• or, there are still a nightmarish number of bugs in the app and further invites are postponed for review until the next release cycle.

Open Beta:

• you feel like your app is sufficiently polished so you release it to the masses as a public beta
• the more objective users start to find and report bugs
• unfortunately, the sheer number of bug submissions bloats the bug tracker so finding bugs becomes sufficiently difficult
• since it's hard to find bugs, duplicates start popping up and the bug tracker bloats even more
• you spend x amount of effort wasted trying to keep the bug tracker clean while also trying to fix bugs in the code
• the less objective users give the app a shot and discover that 'not everything' works perfectly or works in a the way they expected (intuitive)
• the less objective users run off to their little blogs and start making posts like 'OMG WTF srsly, [appName] sucks for reason [x] and reason [y] and [z]
• all the little 'buzz bloggers' trample all over the name of your app because it makes them feel 'empowered' to publicly rant about any/everything
• google indexes all of the blog rants because they contain a lot of indicative keywords related to your product so the top 2 pages that come up when you enter your apps name in google usually involve something along the lines of '[appName] sucks'.

One of the biggest benefits of a 'closed beta' is, you have the ability to control your work load based on how many users you allow and what types of users you allow.

You need an army of 'objective' users to back you up against the 'subjective' users because the ladder group's main purpose in life is to troll the web looking for an app to trash and create a lot of sensational anti-hype over; all in the name of attracting more traffic to their blog.

If you want a really good example of how to successfully run a closed beta look to Google.

• with gmail they had a strictly limited closed beta to start
• they fixed any obvious bugs discovered by the first round of beta testers
• more invites are given out
• then they start collecting new ideas of features to implement in their webapp from the beta users
• they incorporate features while simultaneously fixing bugs
• they dole out more invites
• continue tracking features requests and bug submissions
• when it's sufficiently polished they release it to open beta
• GMail remains in open beta for 3 years

Why? Google is smart. If some random obscure bug pops up 2 years into open beta nobody can really trash google for it because it's still 'beta'. It's like Google's little way of saying, it's good but we're not completely satisfied with just good. Even if they didn't touch the codebase for the last 2 years of the beta, it still gives the impression of 'they're still perfecting it'.

Which leads me to the single most import point of why you'd want to limit the beta...

Once created, you can't change people's perceptions about your product

Watch this, "How to Ignore Marketing and Become Irrelevant in Two Easy Steps" to see what I mean. It's easily one of the most intriguing presentations I've ever seen.

Note: I've personally participated in multiple 'closed' betas. Namely, GMail, Google Wave, Boxee, Songbird, and a few others.