When doing hallway usability tests do most of you make your apps fully or near fully functional? Or do you just make sure the links or flow chain correctly? Or do you just draw on paper and go with that?
I'm would like to test early on a prototype and am trying to find a good balance. But at the same time am worried that some non functional parts might actually not give representative results.
Thanks.
A couple things to remember:
Therefore, if users can see the parts of your UI you are interested in testing and interact with them in a realistic way (e.g., click on buttons and links), you should be able to collect useful data. If some links are dead-ends, that's okay, as long as there's some way for users to recover and continue on. Basically, with prototypes, the "correct" path should work, but it's okay if incorrect paths don't (as long as there's a reasonably quick way to get back on the correct path). Even static storyboards (non-functioning drawings of a UI) can provide you with some information if you ask the right questions, e.g., "What would you do on this screen if you wanted to view your shopping cart?").
I would suggest a couple rounds of usability testing. First on paper, perhaps later on screen, generally throughout the application lifecycle (take an Agile approach to it).
There is a good argument to be made for paper prototypes. When users see a screen, even limited functionality, they may be hesitant to suggest changes since it looks "done."
Make no mistake, it's not trivial to get it all down on paper, but that's where I would start. Probably start with just a section or two of the application. And make sure somebody with good people skills and/or explaining skills is there to walk the user through it. Have a second person on-hand to take notes. Try to ask open-ended questions, etc.
I would make the UI functional, so that the user can really play with it, it will be much better than a static image. People can tell you whether they feel comfortable on the UI.
I would make sure everything in the UI works, or at least takes you to a clear, unambiguous message pointing out that the feature isn't implemented yet.
Showing prototypes to clients with a disclaimer up front about how feature X doesn't work yet will usually be ignored. They'll try out the prototype, click on featuree X and indignantly reply "Feature X doesn't work! This really needs to work in the final version! Why doesn't it work?". The client is confused and unhappy about the product, and it's frustrating for yourself because it overshadows the positive feedback. Besides, you told them it didn't work, why can't they use their imagination to envision how it would work in the final version?
Make it work, be it with a rough version, dummy data, or even a simple message saying "would show results sorted alphabetically now".
Usability tests, hallway or otherwise, only need the functionality that you need to test. In most usability tests, you should go in with specific design questions to answer and develop your prototype to the point where it can answer those questions. For example, if you need to test if users understand your indication of the sort order for a table, all you need is a paper picture of the table showing the sort indication (with the table contents blurred) and ask them how the table is sorted. If you need to test the IA, all you need is a bunch of web pages, empty except for a title, that are linked through the navigation menus.
You only need the pages relevant for the tasks you give your users. If you’re just testing the IA, then you only need the pages on the normative path. If you are also testing error recovery, then you need the pages off the normative path along with the full navigation controls. If you are also testing error detection, then you need content on the pages as well.
You can also simulate functionality when that’s easier to do. For example, in testing if users can figure out how to get a desired sort order, when the user clicks on a non-functioning control for sorting the table, you can say, “Okay, doing that will get you this,” and you take the mouse and select a bookmark that shows the table in the new sort order.
In hallway testing, if users breach the fidelity envelope, you can simply say, “I haven’t made that part yet. Let’s go back to A, and continue from there.” Of course, you should note that the user made a wrong turn in the task you intended for them. I haven’t had any problems with users complaining about non-functional features when I tell them up front it’s an incomplete prototype and we’re only testing the UI for features x, y, and z at the moment.
For low fidelity prototypes, I often call them “mockups” or “drawings” to users rather than “prototypes” to indicate the low functionality. You can put obvious placeholders in for missing content (e.g., “Blah, blah, blah…”, “TODO: Picture of product about here.”). If a user comments on something outside the fidelity envelope (e.g., “This symbol should be red to stand out more”), simply note it, and say that topic is under development (e.g., “Thanks. We haven’t started work on the colors yet. We’re just trying to figure out how to organize the site right now.”).
Usability testing with limited-fidelity prototypes is really necessary for iterative design to be feasible for most projects. Otherwise, you waste too much work developing things that have to be redone.
For a hallway test, I would test with NONE of the functionality implemented.
Test against designs done on a whiteboard or on paper. You'll be surprised at how much you find out in these minimal mockups. And they are very inexpensive to make!
Functional prototypes are for later. If you give your usability subject a functional interface, they are much less likely to question whether you've implemented the right set of features in the first place.