Usability Evaluation Issues in Commercial and Research Systems

Laila Dybkjær, Niels Ole BernsenNISLab, University of Southern Denmark

Hans Dybkjær SpeechLogic™, Prolog Development Center A/S

ASIDE 2005-11-10COST Workshop, Ålborg University

Slides available at www spokendialogue.dk/publications/2005k/ASIDE-2005-11-10.ppt

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

It’s all about design – usable design ...

Users, prompts, modalities, media, ... Usability!

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

Usability in academia and industry

Academia: New and challenging? Focus on advanced systems and new knowledge

Industry: Cost? ROI? Market? Customers? Focus on state-of-the-art and functionality

But they have much to learn from each other Lots of research results to streamline for industry Lots of ”simple systems” questions open to research Gap: EU research visions vs. industrial reality

What can they learn from each other?

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

Three examplesSystem Traffic FAQ NICE HCA

Task / domain

Road traffic information

Holiday allowance information

H. C. Andersen’s life and fairytales edutainment

Purpose Commercial CommercialGov. support

Research

I/O Speech Speech Speech, gesture, 3D graphics

Language Da 50 words Da 500 words En 2000 words

Target Car drivers All employees Children 10-18

Who built it

PDC PDC, NISLab NISLab and 4 other EU partners

Wide range in purpose and complexity

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

Cost and complexity

Academic focus: Prototypes – Industry focus: Final systems

Months

Traffic400 hoursSimple

FAQ4000 hoursComplex

NICE HCA40000 hoursVery complex

0 2 9 2313 36

P1 P22005

F1 F2 2002

F 2005

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

Usability evaluation criteria

Difficult to select right criteria for given system• Is purpose to compare, to investigate,

or to define contract?• To make proper selection, one must know the range

and properties of criteria available

Many usability criteria vaguely defined • E.g. “adequacy” or “sufficiency” of …

Quantifiability often missing• Subjective or qualitative evaluation

New system types require new criteria• Must be clearly defined and operationalised

Standards may emerge, but new needs keep coming

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

Core usability evaluation criteriaSystem Criteria

Traffic Interaction problemsCorrectness

Task and domain completeness

FAQ Interaction problemsCorrectnessTransaction success

Task and domain completeness

NICE HCA Conversation success NaturalnessReasoning capabilities Ease of useError handling

Scope of user modellingEntertainment and education valueUser satisfaction

Clearly different focus in academia and industry

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

Usability evaluation methods

Which one to choose depends e.g. on• Evaluation purpose• Resources (who, time, money)• Stage of development process

Examples of methods• Walkthrough (early)• Focus groups (early, but ok any time)• Wizard-of-Oz (early-middle)• Field test (late)• Heuristic evaluation (best early but also ok later)• User interviews and questionnaires (any time)

Many current practice methods!

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

Usability evaluation methodsSystem Methods

Traffic Walkthrough (using DialogDesigner)Semi-formal WOZ (using DialogDesigner)In-house scenario-based testExpert evaluation of domain information

FAQ Walkthrough (manually) In-house and external scenario-based testQuestionnaire on webMonitored scenario-based lab testsField data analysisExpert evaluation of domain information

NICE HCA WOZ in schools and in museumLab-test of first and second prototypePost-lab-test interviews

Industrial systems need broad range of methods

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

Data and analysisSystem Data Analysis

Traffic Logfiles Problem identification via observation and feedbackLog-based analysis of problems

FAQ Logfiles; trans-criptions; trans-action annotations; questionnaires

Problem identification via ob-servation, feedback from users and domain experts, and ana-lysis of transcribed dialogues

NICE HCA

Logfiles; trans-criptions; topic annotation; Eng-lish evaluation; interviews

Analysis of WOZ and lab test data for design input; analysis of lab tests and interviews to get users’ opinion and develop new criteria

Research more data and analysis needed

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

Electronic model IT tools possible

A lot of knowledge and theory can be made operational

Sketch, prompt design and recording, walk-through, WOZ, document, test, formal properties (coherence, well-formedness, ...), ..., you name it

Spee

chLo

gi c&

N

ISLa

b

ASID

E 20

0520

05-1

1-10

Academia and industry do meet ...

Industrial actions and challenges• Optimise existing processes• Automation (transcription support, annotation, ...)• Use known results and theories• Unknown effects of new technology

Academic challenges• Highly sophisticated technology• New factors to analyse, define, and measure• On-line adaptivity to users’ skills, expertise, …• Investigate troubles with ”simple” systems

... even though they are also different beasts