Journals by RSS
Journals by EmailI conducted some context-sensitive online help usability testing with four users of a Web application. This post discusses the results (without getting overly academic in tone, I hope).
Objective
I wanted to find out whether and how quickly participants could find given pieces of information and answers to specific questions in two versions of a help system described in this post.
Specifically, context-sensitive topics in the first version have a screenshot giving an example of how the screen can be used, sometimes with numbers corresponding to annotations in the text below. The newer versions of extensive help topics had the screenshot hidden in a dropdown hotspot and details contained in dropdown hotspots instead of being annotated. Specifically, the hotspots were formatted as green, underlined text with a right double angle bracket afterward.
Method
I wrote ten tasks for finding certain pieces of information within the context of using a particular screen and then clicking Help. (I realize this takes an assumption that users would take the step of opening help, but this experience actually educated them about using it.)
I looked for participants in one of two main roles in the application and also in a certain demographic that we were concerned about: 50 years old and older. Users in Role A had three tasks in the old version of the help and two tasks in the new version. Users in Role B had two tasks in the old and three tasks in the new.
I explained to each participant that I was testing myself (not them) to see how easy it is to find information in the help. I told them that I would be observing and taking notes; I wanted them to speak out loud so I would know what they were thinking; and it was okay for them to give up if they couldn’t complete the task. I didn’t explain that I was having them test two different versions of the help system so that wouldn’t somehow affect the results.
I ended up skipping a couple of tasks, one old task because I could see that the participant would run into the same trouble that he had on the previous two tasks, so rather than frustrate him or allow him to feel like he was failing, I moved on. I skipped another task because another particular user didn’t have a certain required permission in the system.
Results and Observations
This is the part I was looking forward to, of course. Not being a completely objective party, I hoped that the solution I introduced to visually condense information would prove to be on the right track.
Out of eighteen tasks performed by four participants, twelve were completed successfully. In the old help, tasks were either not completed or were completed after a delay of up to ten seconds. In the new help, tasks that were completed went more quickly.
I found that the screenshots at the top of the topic in the old version were problematic. The participants’ screen resolutions were such that the text didn’t appear below the screenshot, and most of the participants didn’t notice that they could scroll to get more information. They tried interacting with the screenshots or looking for answers on them. One user, because she couldn’t see any help text in the topic, began clicking a set of links I’ve included at the top of every topic. This nav bar consists of links to the FAQ, troubleshooting tips, and so on. Only one of the participants—the one who completed all five tasks successfully—caught on to the annotations and realized that if something was numbered on the screenshot, there must be a corresponding number somewhere below.
Overall, participants had a higher success rate in the new version of the help topics. Because the screenshot was hidden in a hotspot, they could see the help text. The dropspot links provided a way for them to scan the types of information available and choose the one they wanted to learn more about. None of the four had any problem understanding that they could click the dropspots for more information. The different color and underlined formatting were a signal to a couple of the participants, and the double angle bracket was the main signal for another.
Due to the number of tasks completed overall, I saw that while it was still possible to find answers in the old versions of the topics, finding information went more quickly in the new versions.
Participant Comments
After each participant went through the tasks, I showed them the old and the new versions together. In particular, I had used both versions of the topic that had become so bogged down with information (as described in the previous post) in two different tasks. I asked for their overall impressions and comments about the old version and the new and have notes for three of the four.
Participant 1: The old version is “busy; I have a phobia about scrolling through a bunch of stuff to find what I want. I would like it to be more concise, take one question at a time. I like the new version better; I can look at exactly what I want to find out. I already know what the screen looks like, so I don’t necessarily need to know that. I can see that on an initial training that the screenshot is good, and if you’ve never seen it before, it may be good. But for a situation for needing quick help, I need something compact. I already know why I’m coming here. I don’t need to look at the screen and identify all parts of it.”
Participant 3 (who completed all tasks): “I need the title at the top so I know where I am [as opposed to having the screenshot]. Having the numbering on the picture sort of flows. With the green links, I had to poke through. I’m not sure which one’s quicker.”
Participant 4: “The FAQ maybe should have a few more selections. Maybe ten choices [per subject or section], maybe you’d find something along the lines of what you’re looking for.”
Conclusions
In general, the participants reacted more favorably toward the new version. I do probably need something to suggest what parts of the screen the green links correspond to because the participants tended to regard them as topical rather than talking about specific items on the screen. So I’ll have to think about that more. As I mentioned previously, I would also like to have a script that would expand all dropspots so they can be printed.
Related posts (auto-generated):
Ben Reply:
December 24th, 2009 at 9:00 am
The Web app is designed for a 1024×768 resolution, but some of the screens require scrolling. This results in screenshots that are tall enough to make any help text below them fall below the fold. The screen captures are less than 2/3 the size of the actual screen—big enough, I hope, that the users would be able to see the examples, but small enough to take up as little room as possible. The thumbnail idea is a good one. I’ll look into that. Thanks for the suggestion.
There were two main roles I was working with, but there are more granular permissions within one of those roles. (The other role corresponds to a single finance permission.) That particular participant happened to not have the permission needed for that task. Such are the problems with informal testing. I would love to get into spending more time on each project doing usability testing on the docs, so I hope to get better at spotting potential problems like this one.