What is a realistic (but fast) beta versioning schedule?

Question

My team is close to deploying our application, and we're about to go into closed beta with some select customers. I'm wondering what a realistic timeframe would be for producing new beta versions, and how many such cycles we could realistically expect to need before we can call the first version stable enough for release.

The application itself is a medical imaging application, so it absolutely cannot crash or corrupt data. Many users will also be using the software for at least four to eight hours a day continuously, so I expect that normal user errors will be encountered fairly quickly. The application is tied to a specific piece of hardware, and if they have the hardware, they will need this application or the previous version of the application to run their hardware.

Of course, there's also pressure from above to release it now, now, now! and since they pay my paychecks, I'm obliged to follow their instructions, despite whatever misgivings I may have about speedy releases.

I'm thinking that the following scenarios could play out:

1. Two week cycle time. We have a select group of users, say three to five sites, and as they encounter bugs, we fix them. I think that this cycle time is absurdly fast, but I can already feel that it's going to be how the Powers That Be will want to deploy. For this approach, we lock the product to a particular build, and any errors that accumulate we fix in the next release (which could be fifty builds later).
2. Six week cycle time. We have the same select group of users, but that group can grow, and as it grows, we act as in step 1. Not as fast as 1, but certainly more cautious. Problem is, users may get the impression that the product is excessively buggy (if they encounter bugs), and won't have that impression countered until we release another version, at which point they may no longer care. Since there's that lockin to the hardware I mentioned earlier, that impression of bugginess may just translate into mild grumbling rather than lost sales. However, each newer beta version will be just that much more vetted than the last.
3. As fast as bugs are fixed, get fixed versions into the hands of users. We have a build server, we have several testers, and we're pretty quick to respond (you might even say, 'agile'). Are there drawbacks to just giving out bug fixes as fast as we fix a bug, so long as the fix doesn't break some other behavior that the software needs? If we took this approach, would we be doing cycles, or just a beta 'period'?

I realize that these questions vary greatly from user to user, and that something like a Blizzard or Gmail beta period is a bit on the long side. I would still like to get a general feel for how I should respond to management's constant questions about how long it should be in beta.

Answer 1

That seems like a question best answered shortly after your first wave of feedback come in from the initial beta release. If customers are generally happy, and issues are more cosmetic than structural, plan for a shorter beta session. If you find the opposite to be true, prep management to prepare for a longer more in depth beta period.

I don't know the exact specifics of your application, but I would think that a set release schedule will be easier for your customers to keep up with. When a customer does find a high priority issue, it is great to hear that the resolution is in and it will be in the next release in 2 weeks, or even 6 weeks, as long as it isn't months away. 'Patching' unless extremely diciplined can cause more problems than it solves, with customers having odd and vastly different code bases, which will create hard to reproduce issues.

Answer 2

Go with number 1. No doubt, it's probably the best way to go. #2 is obviously not ideal, because of the long cycle time. #3 is tempting, and will cause you pretty much an unbelievable amount of trouble. Here's the reason why: if you deploy fixes to whoever needs them as they need them, you completely lose any rational versioning scheme, and keeping track of which customer has which fixes and which versions is tricky at best.

Answer 3

In our case, we let the customer's feedback dictate our cycle. After releasing the initial beta, we listen to the feedback. Sometimes there is a catastrophic bug and we halt rollout of the beta, fix it and then resume. Othertimes, we collect bugs based on customer importance and then rollout an update once the complaining dies down. Ultimately, the customer's perception is what matters most here (in reference to speed) and so balancing their complaints against what management wants is the tricky aspect. In our case, we generally go with somewhere in the range of 2-4 weeks based on how many and how urgent the bugs are.

Answer 4

I've seen one-and-one release schedules. The first release is planned, bigger, includes features, aybe 6-8 weeks or sometimes more. The second is a planned fixup, 2 week interval. So every 10-12 weeks you have 2 releases.