cse 595, winter 2000 1 january 20 topics (design and evaluation part 2) zadministrivia ynext...

CSE 595, Winter 2000 1

January 20 Topics (Design and Evaluation Part 2)

Administrivia next week’s readings on web (focus: technology);

name tag usability; reading report feedback

Usability Study Guidelines Contextual Design discussion Guest speaker: Domenick Dellino, “What Usability

Can Borrow from Anthropological Methos” More paper discussion:

Ethnography Heuristic Evaluation; Discount Usability Engineering McGrath - Methodology

Project Pitches or Group meetings


Quote of the Week

“You can observe a lot just by watching.” -- Yogi Berra


Another Quote of the Week

“Le mieux est l’ennemi du bien”-- Voltaire


Guidelines for User Testing with Thinking Aloud

Practical study designWritten materialsCarrying out the studyImproving the study

(this material is also linked from the class web page)


Practical Study Design

Reflect on the participants' backgrounds and how they might affect the study

Be aware of problems that arise when experimenters know the users personally

Prepare for the study carefully (avoid last minute panic) Select the tasks carefully to be representative and to fit the

allotted time In general, start with an easier (but not frivolous) task Write down features of the system that are not being

tested as well as those that are!


Practical Study Design (2)

Define the start-up state for the study precisely Define precise rules for when and how users can be helped

during the study Plan the timing and cut-off procedure (if subject gets stuck)

for each part of the study Include reasonable provisions for data collection (e.g.,

notes, tape or video recorder, keystroke capture where appropriate)

Plan data analysis techniques in advance Carry out a pilot study (important but often overlooked)


Written materials

Participant release form (if needed) Questionnaire covering prior experience etc. (if relevant) Introduction to the study for users, including scenario of

use Checklist for experimenters Evaluation survey (if relevant)


Carrying out the study

Let users know that complete anonymity will be preserved Let them know that they may quit at any time Stress that the system is being tested, not the participant Indicate that you are only interested in their thoughts

relevant to the system Demonstrate the thinking-aloud method by acting it out for a

simple task, such as figuring out how to load a stapler, and a computer-related task

Hand out instructions for each part of the study individually, not all at once

Maintain a relaxed environment free of interruptions Encourage users to keep talking using unobtrusive comments

that don't point the user in a particular direction. Debrief each user after the experiment


Improving the study

The pilot study should "debug" the study. This minimizes changes during the study, allowing quantitative data analysis. But improvements may be warranted.

Experimenters' role can be improved Tasks given to participant can be improved Written materials can be improved

CSE 595, Winter 2000 10

Notes on “Contextual Design”

Widely praised in industryThe class rated it highly (average 5.1)An eclectic customer-centered design

methodology — includes ideas from ethnography, participatory design, much empirical work, etc.

Data gathered from customers is the basis for making design decisions

Team and organizational considerations explicitly dealt with

CSE 595, Winter 2000 11

Principles of Contextual Inquiry

The initial data gathering stagePrinciples:

Context: summary vs ongoing experience; abstract vs concrete data

Partnership: master/apprentice model (avoid interviewer/interviewee, expert/novice, guest/host)

Interpretation : check interpretations on the spot (“but won’t it bias the data?”)

Focus: interviewer’s point of view while studying the work

CSE 595, Winter 2000 12

Stages in Contextual Design Methodology

contextual inquiryinterpretation sessionswork modelsaffinity diagramwork model consolidationvisionstoryboardsuser environment designpaper prototyping

CSE 595, Winter 2000 13

Notes on Ethnography

Dictionary definition: “a branch of anthropology dealing with the scientific description of individual cultures”

Many ethnographers reside in the field a year or more, learning the local language, and participating in everyday life, while maintaining some objective detachment (participant observation).

CSE 595, Winter 2000 14

Ethnography — Issues

identification with culture being studiedinformantscultural change resulting from

ethnographer’s presencetechnology: field notes, video, audio

recordings

CSE 595, Winter 2000 15

Relevance for Design

Designers create artifacts for work settings need to understand those settings.

Technology shapes practice designer's world view should not be imposed inappropriately on the users.

Allows us to gain broader understanding of technology in use (see traditional methods)

Joint exploration of technology and work allows both users and designers to participate in new designs.

How Usability Borrows from Anthropological Methods

Usability Research: Anthropology of the Workplace

Domenick J. Dellino

CSE 595, Winter 2000 17

Overview

Traditional anthropological methods Verbal methods Non-verbal methods

Anthropological methods in usability Examples and comparisons

Confounds of each method

CSE 595, Winter 2000 18

Traditional Anthropological Methods — Verbal

Participant observationKey-informant interviewingCollection of life historiesStructured interviews and surveysQuestionnairesRankings and Ratings

CSE 595, Winter 2000 19

Traditional Anthropological Methods—Non-verbal

ProxemicsKinesicsVideotape Research*Content Analysis*

Myths Folktales

Erosion and Accretion

CSE 595, Winter 2000 20

Participant Observation

Trad. Anthropology Anthropologist lives

with the tribe as a member of the community until barely noticed by the natives

Observes and records the culture while participating as a native

Usability Research Contextual Inquiry:

Usability Specialist infiltrates school computer lab as a computer nerd needing remedial training

Observes student’s questions/behavior

CSE 595, Winter 2000 21

Key Informant Interviewing

Trad. Anthropology Anthropologist finds one

native who is willing to tell everything

Informant enjoys special treatment (e.g.: Hershey bars)

Usability Research Participants volunteer

to be recruited for usability studies

Participants are rewarded (e.g.:software)

CSE 595, Winter 2000 22

Collection of Life Histories

Trad. Anthropology Anthropologist develops

rapport with a few individuals

Collects extensive materials about these individuals

Usability Research Heuristic Evaluation: Usability specialist

finds other specialists who understand interface design

Asks them to report all “usability issues” they can find

CSE 595, Winter 2000 23

Structured Interviews & Surveys

Trad. Anthropology Fieldworker

administers formal interview about lifestyle, earnings, expenditures, etc. by going from house to house.

Usability Research Specialist asks

specific questions in the workplace (or lab) about users’ work processes, policies, and practices

Contextual Interview

CSE 595, Winter 2000 24

Questionnaires

Trad. Anthropology Researcher hands out

questionnaires to an informant and returns to collect the answers

E.g.: Informants are asked to record what they purchase during the week

Usability Research Specialist distributes

questionnaires to a group in a lab setting or e-mails survey

Recipients are asked attitudinal and behavioral questions.

CSE 595, Winter 2000 25

Ranking and Ratings

Trad. Anthropology Commonly used to

determine the hierarchical structure of caste, class, or kinship systems

Usability Research Cluster Analysis Participants are

asked to sort cards of menu commands into stacks

CSE 595, Winter 2000 26

Proxemics

Trad. Anthropology Unobtrusive

observations of how close people stand to each other

Usability Research Eye tracker software Tells us where the

user is looking on the screen

CSE 595, Winter 2000 27

Kinesics

Trad. Anthropology Lurking Watching what people

do Observing the people

one touches, grooms, hits, holds

Usability Research Counts of clicks and

keystrokes Counts of “mistakes” Recording of success

rate

CSE 595, Winter 2000 28

Erosion & Accretion

Trad. Anthropology Archaeology Study of paths Studies of middens,

fireplaces, burial grounds, & tool making sites

Usability Research Use of “instrumented

versions” Investigation of query

& error logs, Web cache, trash

The “Lame” button F1 key wear

CSE 595, Winter 2000 29

Confounds of Each Method-1

Participant Observation Heisenberg Uncertainty Principle

Key Informant Interviewing/Life Histories Sampling—Informants may not be

“representative”

Structured Interviews & Surveys “Performance anxiety” Memory may be flawed, suspicion

CSE 595, Winter 2000 30


Questionnaires Literacy, focus, desire to show favor

Ranking and Ratings Task requires sophisticated thought Data analysis is aggregate (non-parametric) Validity of interpretation (are we measuring

what we think we’re measuring?)

CSE 595, Winter 2000 31


Proxemics Ethnocentric/design-centric interpretation

Kinesics Question of learning style: “I like to try

everything first.”

Erosion & Accretion “Whose data is it, anyway?”

CSE 595, Winter 2000 32

…for Reflection

What value does ethnography add?When should it be conducted?Is it cost effective?

CSE 595, Winter 2000 33

Ethnography — Ethics

anthropologists must be open about the purpose, potential impacts, and source of support

primary responsibility to people and animals with whom researchers work

responsibility to scholarship and scienceresponsibility to the public

American Anthropological Association link:http://www.ameranthassn.org/committees/ethics/ethics.htm

CSE 595, Winter 2000 34

Notes on J. Blomberg et al, “Ethnographic Field Methods”

Rating: 4.3 Why Observe? Why not just ask?

ideal vs. manifest behavior "What people say and do are not the same thing." People have a great deal of tacit knowledge that they cannot

verbalize.

Observational role: Unobtrusive Observer (aka Observer Participant).

This is difficult or inappropriate in many settings. Participant Observer

+access, +first-hand experience, +-point-of-view issues, -logistical problems of recording

CSE 595, Winter 2000 35

Blomberg (2)

In reality, the ethnographer doesn't assume either of the above extremes, but moves back and forth along the continuum between pure observer and pure participant.

CSE 595, Winter 2000 36

Blomberg (3)

What do we focus on? Event focus - meetings, seminars, ceremonies. Person focus - "a day/week/year in the life

of..." Place focus - receptionist's desk, printer room,

etc. Object focus - life history of a document,

transaction, etc.

How do you know you are finished? When you're no longer suprised by what you're

seeing!

CSE 595, Winter 2000 37

Blomberg (4) — Techniques

Notes: + inexpensive, interpretive, flexible (sketches,

etc), holistic - interpretive, can be hard to reconstruct, "low

bandwidth," lots of work for observer

Audiotape: + inexpensive, "medium bandwidth," less work

for observer - limited slice of activity (audible/verbal only),

requires some audible actions to be useful, difficult to reconstruct

CSE 595, Winter 2000 38

Blomberg — More Techniques

Videotape: + "high bandwidth," can make detailed reconstructions

(content log), advantages of audio, less work for observer, unobstrusive, somewhat holistic

- limited slice of activity, expensive, building content logs can be exhausting, coordinating multiple videos difficult, activity space/lighting might not be suitable for filming

Event Logs: (computer capture of input events) + "high bandwidth," can make detailed reconstructions, can

get good statistics about failures/difficulties, time sensitive - computer only, emphasizes human-machine dyad view –

human is viewed as mere source of input events anti-holistic

CSE 595, Winter 2000 39

Linking ethnography and design

Ethnographer reports findings to designers

Ethnographic study is undertaken by team of ethnographers and designers

Full participation in designing by ethnographers, designers, and users

Issues: Whose "side" is the ethnographer on? Who "owns" the result (knowledge) of

ethnographic research?

CSE 595, Winter 2000 40

Notes on Jacob Nielsen, “Guerilla HCI”

average rating: 5.1

Techniques: scenarios simplified thinking aloud heuristic evaluation

CSE 595, Winter 2000 41

Notes on Jacob Nielsen, “Heuristic Evaluation”

average rating: 4.3

Heuristic evaluation: have a small set of evaluators examine an interface and judge its compliance with recognized usability principles (the “heuristics”).

Evaluators work independently. (Can compare results afterwards.)

Output: a list of usability problems with reference to the heuristics.

Comment: doesn’t require working with eventual real users.

CSE 595, Winter 2000 42

# of Evaluators vs Problems

CSE 595, Winter 2000 43

Cost-Benefit Ratio

CSE 595, Winter 2000 44

Ten Usability Heuristics (from Nielsen)

Visibility of system status Match between system and the real world User control and freedom Consistency and standards Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize, diagnose, and recover from

errors Help and documentation

http://www.useit.com/papers/heuristic/heuristic_list.html

CSE 595, Winter 2000 45

Usability Heuristics Applied to the Web

Highly recommended:

Keith Instone’s discussion of applying these to the web

(http://webreview.com/97/10/10/usability/sidebar.html)

http://webreview.com/97/10/10/usability/sidebar.html



CSE 595, Winter 2000 46

Notes on Joseph McGrath, “Methodology Matters”

Ratings: binary (five 1’s, three 6’s). Average 3.5

I’ve received very positive comments on this paper from social scientists

science vs. design and engineering; discount usability engineering

CSE 595, Winter 2000 47

McGrath: 3 domains

content that is of interest (e.g. grad students using Powerpoint to prepare a job talk). This is the substantive domain.

ideas that give meaning to the content (e.g. "able to prepare slides, frustrated" — i.e. task-related descriptions, affect-related descriptions, etc). This is the conceptual domain.

techniques or procedures for studying the ideas and content (ethnography, laboratory experiment, field study, heuristic evaluation). This is the methodological domain.

CSE 595, Winter 2000 48

McGrath: techniques

Techniques for manipulating features of the research situation: giving instruction Imposing constraints Selecting materials Giving feedback Using experimental confederates

CSE 595, Winter 2000 49

McGrath: desirable features of a research strategy

generalizibilityprecisionrealism

We can't maximize all of these at the same time!

We can have multiple studies, and ask "does study A support study B?"

CSE 595, Winter 2000 50

McGrath: Some key concepts

BaseratesCorrelationsCausality

cse 595, winter 2000 1 january 20 topics (design and evaluation part 2) zadministrivia ynext...

Documents

pilot study important

studythe pilot study

relevant cse

improved cse

topics design

studylet users

experiment cse

improved tasks