Tips for beta testing a game?

Question

Hello!

We are finishing a game that we would like to beta test. We don’t have the time nor the budget to do a large scale testing, we will simply show the game to some of our friends. (I hope that we can still gather something useful of it.) In the Age of Empires post mortem (Postmortems from Game Developer) I found the following:

A public beta test of AOE was held in August 1997, but we didn’t come near to exploiting the full potential of it. We were too close to the end of the project to make any game play changes, despite the wealth of useful feedback we received. Manuals were already set to be printed, and most of the design had been set in stone. All we could really do was fix any bugs that were found.

This is a good point. Are there any other good advices? What should (not) we tell the people beforehand? What should we look for?

Answer 1

Certainly you need to go beyond friends. The problem with only testing with friends is that they'll want you to succeed, and thus tell you it's good. (I mean look at all the people encouraged to try out for American Idol for crying out loud!)

If you can get some sort of private beta going, that would be ideal. Because you do not have the budget for anything large scale, it will have to do. If you can get a decent enough population you should be able to get better feedback than from just friends. I wouldn't worry too much about diversifying the demographics, but certainly target people who would actually be interested in whatever sort of genre your game is in (I'm going to assume RTS based on the example you've shown).

Answer 2

My suggestions are to get a website with a forum set up somewhere so that members of the public can apply for beta access and are able to provide you with feedback. At some point, you'll probably also want someone to regularly trawl the forum for bug reports and read the general feed back. The bug reports can be added to a bug tacking database so that you have a handle on the issues being raised by the community and the general feedback can be used to provide a regular report to the rest of the team as not everyone will have time to read every single post.

Once you have a community built up and the processes in place for gathering feedback, you should start to think about what you want to actually test.

If you have specific game play areas you are unsure about, ask the community directly about those areas. If you are looking to test the network code or scalability of servers, organise regular test nights (no more than 1-2 hours though) with a specific set of tasks to perform. This will produce a ton of feedback the next day, but you'll find it mostly focused on the areas you asked people to test saving you the trouble of having it randomly scatter through the forums.

Generally, people will want to help improve the game and getting the community involved earlier will not only get you important feed back from the people who will be playing it when it's finished, but also gives the community a sense of ownership. It's their game and they've helped make it better. The more you can interact with the community in some way, the more you will get from them when it comes to organised tests.

Answer 3

You can discover quick facts about your game interface by doing some hallway testing; just find some random people, ask them to try the game, and watch them playing (you can record the desktop, or record the player from the back).

As @AlbertoPL said, you may want to find some dedicated players, for advanced feedback on

• learning curve,
• how addictive it is,
• how the game is behaving in the long run,
• how rewarding it is,
• and how it may be abused.

Don't forget to have an issue tracking system installed and easy to use; you can use it to track defects, ideas, and general feedback. I recommend trac, because it has a wiki and it's easy to use.

Great tips @Midpoint for setting up a community.

Answer 4

You might like to read this Wired article about the testing they did for Halo 3. I know you don't have that kind of budget but it might inspire a few ideas or alternative ways to look at your game. One bit I found interesting was the logging of kills/player deaths and time spent in certain areas of the maps so they knew about where people got stuck or if a map was biased in any team play.

Answer 5

Get a wide range of people to test your game. Things that some find obvious are not close to apparent for others.

Also, recruit family members and friends, as well as people with no motive to say "your game rocks!" no matter what they actually think.

Answer 6

One very important aspect is that you should actually watch people play to see what they're really doing, rather than just asking them afterwards what they think about the game.

This will give you a much better idea of where they have problems, i.e. the real learning curve (which is often not something people notice themselves) and what should be made easier ot taught explicitly in a tutorial. It will also show you problems in the user interface, which people may not report when tey've found a workaround; they just remember the UI to be inconvenient.

I'm currently also involved in game testing (a jump'n run game) and an intersting anecdote is that testers who had very little experience playing computer games didn't realize that the character could move sideways while falling. The funny thing is that some testers actually did move the character when falling into deadly situations and often avoided dying that way - but then did not consciously remember that ability when it was required to get somewhere and falling straight down would not result in dying.

The conclusion was that there needed to be a situation early in the game where it was absolutely obvious that the only way to continue was to move while falling, in a situation where falling straight down was not harmful.