Holistic Usability MeasureVUI Design: Philosophy and Practice

2 of 18

VUI Design: Science vs. Art

Our session was described as “... dives deep into some of the science-versus-art issues in dialog design.”

3 of 18

Art & Science in VUI Design

VUI Design is both an Art and Science

4 of 18

Art & Science in VUI Design

Art in VUI Design Creative uses of text/speech

More than 1 way to say something Creative uses of audio “Color” in the interface

Voice talent Mood of the interface

5 of 18

Art & Science in VUI Design

Science in VUI Design Usability

Measurable metrics Human factors

Short term memory Software engineering

Reuse Development

6 of 18

Art & Science in VUI Design

What really matters?

7 of 18

Usability! “If you could build a system

that resulted in world peace, but no one could use it ... it would be useless. Usability matters.” – Gilbert

However, “there has been no generally agreed-upon method of measuring usability, which makes it difficult to compare among different findings.” - Polkosky

Holistic Usability Measure

9 of 18

VUI Goals

What is the goal of your VUI?

Who decides the primary goal of the VUI? Designer Client Manager

10 of 18

VUI Goals

Did you build the right VUI? Who decides?

How do you know if your VUI is better than another that does the same tasks?

You must test it!

11 of 18

Background

During requirement analysis, before design Information design; what is to be built What is the VUI suppose to accomplish? Who are the users? Etc.

These are typical, but most people miss Usability metrics These are typically an afterthought As a result, usability gets cut

12 of 18

Background

Quantitative Metrics Speech recognition accuracy Task completion time

Subjective Metrics User friendliness Ease of use

13 of 18

Background

Which metrics are most important? It depends on your VUI Decide how you will test your VUI before you

design it

Given several different metrics How do you measure the overall effectiveness

of the VUI?

14 of 18

Holistic Usability Measure

HUM Assign each metric a weight (1 – 99%) based on

its importance

Design and implement the VUI

Evaluate the VUI Convert all metrics to a similar numerical scale

15 of 18

Holistic Usability Measure

HUM = W1*Metric1 + W2*Metric2 + ... + Wn*Metric n

where W1 + W2 + ... + Wn = 1

and 0 ≤ W1, W2, ..., Wn ≤ 1

HUM yields a percentage measure rating the VUI’s usability.

16 of 18

Conclusions

VUI Design is both an Art & Science However, what really matters is usability

The goodness of your VUI can only be determined through usability testing Otherwise, you are guessing

Testing is essential!

17 of 18

Conclusions HUM is a tool for testing

It reports usability with respect to the VUI goals

Multiple ways to design same VUI. Which one is the best? HUM is ideal for comparing more than one implementation of the

same VUI

HUM is easy to use and it makes since

Recall, “there has been no generally agreed-upon method of measuring usability, which makes it difficult to compare among different findings.” – Polkosky (SpeechTECH Magazine November/December 2005)

Thank You

http://www.HumanCenteredComputing.org