UX Resources

Interactive Voice Response (IVR) Case Study

Testing Your Telephone-Based e-Commerce Support (continued)

published in The Journal of Electronic Commerce, Volume 12, Number 2

General Usability Questionnaire

Note: The information you provide is kept completely confidential and no information is stored on computer media that could identify you as a person.

We would like to get your general impressions of the usability of the telephone interaction that you used today. Please take a few minutes to answer the questions. Thanks for your help in understanding telephone interaction.

Please indicate your agreement or disagreement with the statements in the left column by circling the appropriate number in the right column. If you are undecided or the question appears irrelevant, then circle the middle number (4). (Circle one rating number per statement).

1. This telephone menu was easy to learn.	1	2	3	4	5	6	7
2. Finding the options to complete the tasks was easy.	1	2	3	4	5	6	7
3. I feel in command of this telephone menu when I am using it.	1	2	3	4	5	6	7
4. There are too many steps required to get something to work.	1	2	3	4	5	6	7
5. Working with this telephone menu is satisfying.	1	2	3	4	5	6	7
6. The terminology is inconsistent with the terminology I know.	1	2	3	4	5	6	7
7. The telephone menu occasionally behaves in a way that can't be understood.	1	2	3	4	5	6	7
8. It is easy to make the telephone menu do exactly what I want.	1	2	3	4	5	6	7
9. The telephone menu seems to disrupt the way I normally do my work.	1	2	3	4	5	6	7
10. The instructions and prompts are helpful.	1	2	3	4	5	6	7
11. It was easy to find information I needed in this telephone menu.	1	2	3	4	5	6	7
12. Learning how to use this telephone menu was difficult.	1	2	3	4	5	6	7
13. There is never enough information when it's needed.	1	2	3	4	5	6	7
14. The telephone menu has a very attractive presentation.	1	2	3	4	5	6	7
15. Error messages are not adequate.	1	2	3	4	5	6	7
16. I have to look for assistance most times when I use this telephone menu.	1	2	3	4	5	6	7
17. These telephone menu features will make me more successful getting what I need.	1	2	3	4	5	6	7
18. I will never learn to use all that is offered in this telephone menu.	1	2	3	4	5	6	7
19. This telephone menu is really very awkward.	1	2	3	4	5	6	7
20. There is too much to learn before one can use this telephone menu.	1	2	3	4	5	6	7
21. It is easy to hear what the options are at each stage.	1	2	3	4	5	6	7

Comment Two subjects exhibited "experimenter demand" effects-they gave extraordinarily high ratings for many of the questions, although each passed only 4 of 10 tasks. Their global ratings were quite high at 6.6 and 5.9 out of a possible 7. We investigated any demographic correlate that would motive the high rating – such as English language shortcomings, and consequent misunderstanding of the situation. While one of the subjects indeed was "ESL" (English as a second language), the other subject spoke English as a first language. We mention this aspect of our data collection to illustrate issues that may arise in your own testing. "Experimenter demand" effect is a known phenomenon of psychological testing in which the subject attempts to give responses to questions they feel the experimenter wants, regardless of their personal experience. While we cannot say for sure this is the case, we had data from enough other subjects to establish a reasonable range of response patterns.

Step 7. Next Steps

Introduction Examination of subject comments reveal these sources of confusion ...

menus that were too lengthy for easy comprehension
unclear vocabulary terms
inconsistent terminology organization of the menus
unclear categories
too many steps (menus too deep)

Furthermore, in 40% of the 120 events failure occurred on the first menu, while slightly fewer fails occurred across the other 4 levels of menu. The immediate presentation of difficulties – at the first menu – indicates a need to focus on general design principles including category definition and minimizing demand on the caller's short-term memory.

Menu Level	# of Failed Events	% Failure
1	56	40%
2	9	6.4%
3	25	17.9%
4	8	5.7%
5	4	2.9%
Total	102	72.9%

Sources and Types of Design Help

The human factors literature provides excellent guidance on issues of IVR design. Much work has been accomplished by Vrizi and Resnick in the context of their research for GTE Laboratories. See their articles listed in the bibliography. In particular, they report a technique of "skip and scan" that we have found very useful when a caller is faced with six to 15 menu options. They have also conducted usability studies comparing skip and scan to a more conventional method. They found that skip and scan was superior for both novice and expert users in all but the first few trials. They used 36 tasks and two IVR applications for the tests.
Some managers may question the value of IVR for their business. A study of public attitudes about IVR as well as telephone answering machines indicate that attitudes varied strongly by age (Kaatz, Aspden, and Reich, 1977). But more importantly, they found that "information rich" persons were not any more positively inclined to IVR than the "information poor." The most significant predictor of IVR acceptance was the quality of one's most recent experience with the technology. This conclusion implies that good IVR design begets positive acceptance.

What do callers want in IVR voice quality? In a study of 84 personality traits, more than 50% of the 50 subjects wanted the following (Chin, 1996):

Practical (78% of subjects)
Intelligent (76%)
Courteous (72%)
Efficient (68%)
Straightforward (60%)
Methodical (54%)
Sophisticated (50%)

Other top scoring qualities were selected by more than 28% of the subjects

Progressive (44%)
Alert (30%)
Scientifically minded (30%)
Imaginative (28%)

To use this data, you could create a "Voice Quality Questionnaire" similar to the "Satisfaction Questionnaire." Use the 11 adjectives listed above, and let your subjects rate sample voice messages.

Standards

The benefits of a standard approach to designing IVR can be obvious to managers faced with large-scale development. Such challenges require coordination among numerous developers and consistency between various menus and even projects. The Human Factors and Ergonomics Society (HFES) has coordinated development of U.S. standards with the international ISO 9241 human-computer interaction standard. The HFES effort is called ANSI/HFES 200 of which section 9 covers Interactive Voice Response, among other voice design topics (Blanchard, 1997).

Standards can be simple and easy to follow – in fact, they will be rejected by developers without such crafting. Following is a sample list of "call flow structures" that meet the needs of a given organization. Our company, HFI, has found this an excellent approach to designing IVR standards. Each organization requires different templates of such call flow structures that readily guide the development staff. Upon selecting a task, the developer need only attend to the recommended template that describes the method.

If the caller's task is...*	Then use a...
Navigating between menus	Selection widget
Selecting one from a list Less than seven items Between six and 15 items More than 15 items	Pick One Scan and Pick One Enter Alphanumeric / Direct Select or Search for Selection
Selecting more than one item from a list	Pick Many
Selecting item(s) from a list by name	Search for Selection
Entering alphanumerics	Enter Alphanumeric/ Direct Select
Transferring shares between accounts or purchasing new shares with funds in current holdings	Exchange
Purchasing shares	Purchase
Redeeming shares	Redeem
Playback of data messages with multiple, addressable parts	Segmented List Playback
Playback of data messages with one item or single message	Nonsegmented Playback
Escaping from current activity	Control/Trap Menu
List management	List Editing
Accessing the voice system	Greeting Menu or Greeting Message

*Thanks to John Ims, PhD, and Gary Dorst, PhD for creating this table.

Bibliography

Blanchard, H.E. (1997). Standards: HCI standards in the United States. SIGCHI Bulletin 29(2)

Chin, J.P. (1996). Personality trait attributions to voice mail user interfaces. CHI '96 Electronic Proceedings, Short Papers.

Gardner-Bonneau, D. J. (1992). Human factors problems in interactive voice response (IVR) applications: do we need a guideline/standard? Proceedings of the Human Factors Society 36th Annual Meeting, 1, 222-226.

Kaatz, J., Aspden, P, and Reich, W.A. (1997). A national survey of opinions about voice response units and telephone answering machines: three surveys and a framework. Behavior and Information Technology 16(3), 125-144.

Resnick, P. and Virzi, R. A. (1992). Skip and scan: cleaning up telephone interfaces. Proceedings of ACM CHI '92 Conference on Human Factors in Computing Systems, May 3-7, 419-426.

Resnick, P. and Virzi, R.A. (1995). Relief from the audio interface blues: expanding the spectrum of menu, list, and form styles research contributions. ACM Transactions on Computer-Human Interaction 2(2), 145-176.

Schaffer, E. M. (1998). How to design effective Graphical User Interfaces – 3 day seminar. Human Factors International, Inc., Fairfield, IA, 52556.

Virzi, R. A., Resnick, P. and Ottens, D (1992). Skip and scan telephone menus: user performance as a function of experience. Proceedings of the Human Factors Society 36th Annual Meeting, 1, 211-215.

Virzi, R.A., Sokolov, J.L., Karis, D (1996). Usability problem identification using both low- and high-fidelity prototypes. CHI '96 Electronic Proceedings, Papers.

Top