Usability

2

Plan for today

Chapter 10 + some examples • What is usability testing • Usability testing versus traditional research • Types of usability testing • User-based testing • Questions for discussion

3

Before we start…

• Two students together • One student should find out time and place

for the next INF2260 lecture • Another should measure the time, note the

answer, find out if this was difficult

4

What is usability?

"The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use." (ISO 9241-11)

5

Usability dimensions

– Effectiveness - can users complete tasks, achieve goals with the product, i.e. do what they want to do?

– Efficiency - how much effort do users require to do this? (Often measured in time)

– Satisfaction – what do users think about the products ease of use?

6

….which are affected by

– The users - who is using the product? e.g. are they

highly trained and experienced users, or novices? – Their goals - what are the users trying to do with

the product - does it support what they want to do with it?

– The usage situation (or 'context of use') - where and how is the product being used?

7

Usability cont. • Usability is composed of:

– Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?

– Efficiency: Once users have learned the design, how quickly can they perform tasks?

– Memorability: When users return to the design after a period of not using it, how easily can they re establish proficiency?

– Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?

– Satisfaction: How pleasant is it to use the design? (Nielsen,1993 )

8

What is usability testing?

• Involving representative users, representative tasks, representative environments – Testing paper prototypes – Wizard of Oz – Testing working version of a system before it is

released – Testing working system – Different devices (smart phones, laptops…)

9

Goal of usability testing

• Improve the quality of an interface by finding flaws in it – Flaws that cause problems for the majority of

people (not preferences)

• What works fine (keep it)

10

Kjdsgklglohofho fhihfdoho

Scope of usability testing

•  Usability testing is a general term that could mean: – Having hundreds of users test interfaces, in

different treatment groups – Sitting next to a total of 3 users and watching

as they attempt different tasks – The reality is usually somewhere in between

•  The best usability testing is the one that actually takes place – Be flexible, be reasonable, be practical

Usability versus research?

• Approach similar to one used in classic research • Different goals • Usability testing is involved in building a

successful product (resources, time, trade-offs) – Practical, large impact – Optimize interface after each iteration

• Usability study – Research study - Research on usability methods

12

Is it traditional research?

•  Is usability testing considered to be “research”?

•  The historical roots of usability testing and experimental research and ethnography are similar

•  Observation, key logging, click-stream analysis, quantitative measurement such as task and time performance, anonymity of participants are all part of usability testing

Is it traditional research?

•  The end goals of usability testing and classical research are different!

•  Usability testing: – An industry-based approach to improving

interfaces – Little concern for using only one approach or

a theoretical point of view – Building a successful product using the

fewest resources, the fewest risks, in the shortest amount of time

Is it traditional research?

•  Practices that would be unacceptable in experimental design, such as modifying the interface after every user, are acceptance in usability testing

•  Usability testing is not used to generalize to other interfaces, situations, or user groups

•  Many companies keep their usability testing results confidential

A comparison Classical Research Usability Testing

Experimental design- isolate and understand specific phenomena, with the goal of generalization to other problems

Find and fix flaws in a specific interface, no goal of generalization

Experimental design- a larger number of participants is required

A small number of participants can be utilized

Ethnography- observe to understand the context of people, groups, and organizations

Observe to understand where in the interface users are having problems.

Ethnography- researcher participation is encouraged

Researcher participation is not encouraged in any way

A comparison

Classical Research Usability Testing

Ethnography-longer-term research method

Short-term research method

Ethnography or experimental design- used to understand problems and/or answer research questions

Used in systems and interface development

Ethnography or experimental design-used at earlier stages, often separate (or only quasi-related) from the interface development process

Typically takes place in later stages, after interfaces (or prototype versions of interfaces) have already been developed

Used for understanding problems Used for evaluating solutions

Research ABOUT usability testing

•  So, usability testing IS different from traditional research

•  But, there is also research ABOUT usability testing, with research questions such as: – Which methods are most effective – Which methods are most cost-effective – How many users are optimal

Analysis Early Design Late Design Implementation Deployment

Goa

lsM

etho

dsA

rtifa

cts

Explore design space

Paper prototypesTask scenarios

Refine selected design

Empirical study

UI guidelines

Functional test

Functional prototype

Implement and integrate

Empirical study

UI guidelines

Functional test

Functionalinterface & system

Gather requirements for next release

Field studyCritical incident walkthroughsInterviews and surveys

Usability report

Contextual inquiry

SurveysObservations

InterviewsTask analysis

User descriptions

Affinity diagramsDesign brief

Task descriptions

Identify:Usability goals

ContentUsers & tasks

Sketching & brainstorming

Cognitive walkthroughGOMS

Heuristic evaluation

Action analysis

Task  Centered  User  Interface  Design  

Usability engineering

Activities aiming to improve the ease of use of an interface • Expert-based testing (usability inspection) • Automated testing (usability inspection) • User-based testing (usability testing)

14

Expert-based testing • Structured inspections done by interface experts • Before tests with users • Confusing wording, inconsistent layout, obvious flaws • Heuristic review

– Compare interface with the rules • Consistency inspections

– Series of screens or web pages inspected • Cognitive walkthrough

– Experts perform the tasks (high-frequency and important/seldom)

• Guidelines review – Web Content Accessibility Guidelines

15

Heuris5c  evalua5on  

•  Heuristic evaluation enables designers to evaluate an interface without users –  inspection, guided by a set of guidelines

•  Economical technique to identify usability issues

early in the design process – no implementation or users required – can be performed on existing interfaces

•  Briefing –  teach to evaluators; ensure

each person receives same briefing.

–  become familiar with the UI and domain

•  Evaluation period –  compare UI against heuristics –  spend 1-2 hours with interface;

minimal 2 interface passes –  take notes

•  Debriefing session –  Prioritize problems; rate severity –  aggregate results –  discuss outcomes with design/

development team

Jakob  Neilson  Sa.sfac.on  Efficiency  Learnability  Low  Errors  Memorability  

Ben  Shneiderman  Ease  of  learning    Speed  of  task  comple.on    Low  error  rate    Reten.on  of  knowledge  over  .me    User  sa.sfac.on  

1.  Validity of system status - Are users kept informed about what is going on? - Is appropriate feedback provided within reasonable time about a user’s action?

2.  Match between system and the real world - Is the language used at the interface simple? - Are the words, phrases and concepts used familiar to the user?

3.  User control and freedom - Are there ways of allowing users to easily escape from places they unexpectedly find themselves in?

4.  Consistency and standards - Are the ways of performing similar actions consistent?

5.  Help users recognize, diagnose, and recover from errors - Are user messages helpful? - Do they use plain language to describe the nature of the problem and suggest a way of solving it?

6.  Error prevention - Is it easy to make errors? - If so, where and why?

7.  Recognition rather than recall - Are objects, actions and options always visible?

8.  Flexibility and efficiency of use - Have accelerators (i.e. shortcuts) been provided that allow more experience users to carry out tasks more quickly?

9.  Aesthetic and minimalist design? - Is any unnecessary and irrelevant information provided?

10.  Help and documentation - Is help information provided that can be easily searched and easily followed?

•  These heuristics are too general in some cases

•  What about web pages? Ambient displays? Ubiquitous computing applications?

•  Strong need for heuristics that are tailored to specific products.

•  High quality content •  Often updated •  Minimal download time •  Ease of use •  Relevant to users’ needs •  Unique to the online medium •  Netcentric corporate culture - Nielsen 1999 for commercial websites

16

Severity  ra5ngs  0 – this is not a usability problem 1 – cosmetic problem only 2 – minor usability problem 3 – major usability problem 4 – usability catastrophe; imperative to fix Combination of frequency and impact

Pros  and  Cons  of  HE  

•  Pros – Very cost effective –  Identifies many usability issues

•  Cons

–  relies on interpretation of guidelines – guidelines may be too generic – needs more than one evaluator to be effective

Cognitive walkthrough involve simulating a user’s problem-solving process at each step in the human-computer dialog, checking to see if the user’s goals and memory for actions can be assumed to lead to the next correct action. –Nielsen and Mack, 1994

Why we use it

•  Cognitive walkthrough enables a designer to evaluate an interface without users –  a designer attempts to see the interface from the perspective of a user

•  Low-investment technique to identify task-related usability issues early on –  no implementation or users required –  can be performed on existing interfaces

•  Identify task-related problems before implementation –  invest a little now, save a lot later

•  Enables rapid iteration early in design –  can do several evaluations of trouble points

•  Evaluations are only effective if your team –  has the right skill set –  wants to improve the design, not defend it

Walkthrough Basics

•  Imagine how well a user could perform tasks with your low-fidelity prototype

•  Manipulate prototype as you go –  evaluate choice-points in the interface –  evaluate labels or options –  evaluate likely user navigation errors

•  Revise prototype and perform again

When to do the Walkthrough

•  Have a low-fidelity prototype of the interface •  Know who the users are •  Have task descriptions •  Have scenarios designed to complete the task

–  you have a “functional” paper prototype

•  Viable once the scenario and paper prototype are complete

What You Need

•  Task descriptions

•  Low-fidelity prototype with enough “functionality” for several tasks

•  Evaluation team: –  design team –  design team and users together –  design team and other skilled designers

For Each Action in a Task:

•  Tell a story of why a user would perform it

•  Critique the story by asking critical questions –  is the control for the desired action visible? –  will a user see that the control produces the

desired effect? –  will a user select a different control instead? –  will the action have the effect that the user intends? –  will a user understand the feedback and proceed correctly?

More on Questions

•  Some extra questions can help –  what happens if the user is wrong? Is there feedback to correct the

error? –  how would a user of <interface> react here?

•  Questions help you see problems –  they are a focus, not a blindfold

Walkthrough Pros

•  Easy to learn •  Can perform early in the design process •  Questions assumptions about what a user may

be thinking •  Helps identify controls obvious to the designer

but not a user •  Helps identify difficulties with labels and prompts •  Helps identify inadequate feedback

Walkthrough Cons

•  Is diagnostic, not prescriptive •  Focuses mostly on novice users •  Designers must put themselves in users mind •  Focus specifically on task-related issues •  The interactions are slower and not real •  Does not provide quantitative results

•  A useful tool in conjunction with others

Exercise

•  Compare and contrast a cognitive walkthrough with a heuristic evaluation

•  Apply Heuristic Evaluation to an existing electronic voting interface http://www.vtintl.com/new/demo/

Automated usability testing

• Software that compare interface with the guidelines

• Produce report and/or fix the code • Manual check often needed • <alt> tag (alternative for graphics) but not if the

text is appropriate – 'picture here'

• Number of fonts, avg. font size, deepest level of a menu

• Software applications: RAMP, InFocus

17

User-­‐based  tes.ng  Ø  Testing is used to support a decision

–  For example, “this design change is going to be better for users”, or “design A is better than design B”

Ø  Research to used to test a hypothesis based on a theory –  Smoking increases the likelihood of developing cancer

Ø  Usability testing is generally done with small samples (mostly do to the cost associated with any alternatives)

Ø  Statistics are used to provide a way relate the small sample tested to the larger population Ø  All statistical analysis assumes the data obtained is valid and reliable Ø  Validity is the degree to which the results of a research study provide

trustworthy information about the truth or falsity of the hypothesis* Ø  Internal validity refers to the situation where the “experimental treatments make a difference in this specific

experimental instance” (from Cambell, D.T. & Stanley, J.C. (1963) Experimental and Quasi-experimental Designs for Research

Ø  External validity asks the question of “generalizability”

Ø  Reliability is the ability of a test to show the same results if conducted multiple times –  Test-retest reliability –  Repeatability –  Reproducibility

User-based testing

– Select representative users – Select the setting – Decide what tasks users should perform – Decide what type of data to collect – Before the test session (informed consent, etc.) – During the test session – Debriefing after the session

18

When to test – Users perform tasks (early or later in the

development) – Formative testing

• Low-level fidelity prototypes • How the user perceives an interface component? • Low-cost of paper prototypes, users comfortable to criticize

– Summative testing • Evaluate the effectiveness of specific design choices • High fidelity prototype

– Validation test • Before release, compare to benchmarks

– Formal approach 19

Usability evaluation Talhonia

time

20

Usability evaluation Talhonia – cont.

21

How many users?

– 5 users will find approximately 80% of problems – 7 for small projects, 15 for medium-large projects – Goal to find the major flows , that will cause the

most problems and must be fixed – How many users can we afford? How many users

can we get? How many users do we have time for?

22

Location

– Lab – Portable usability lab – Remote studies (time, place)

23

Tasks

• Clear; no need for further explanation • Tested • One clear answer – way to do things • Tasks that are performed often • Critical tasks – logging • No private, financial information • Must be clear how to go to the next task

24

Oppgave 4: Du er hjemme og du ser NRK Nett-TV på din PC ( http://www.nrk.no/nett-tv ). Se på episoden 10:12 av Matlyst og svar følgende spørsmål: • Hva heter kvinnen som er kledd i lilla genser

og brunt forkle? • Når en potet frøs om vinteren, hva ble den

da?

25

Measurement

– Task performance – Time performance – User satisfaction (validated survey tool) – Average time to recover from an error – Time spent using help – Number of visits to the search feature – Time spent on specific web page – Typing speed – Qualitative data

0

5

10

15

20

25

30

35

40

45

1 3 5 6 7 10 11 12 13 17 18 20 21 23 24 25 26

Unified Library Application - QuitUnified Library ApplicationTitle InformationReturnItemReserve TitleLend ItemInsert Title WindowFind TitleCompilation

Subject_id 15 Task 2

Sum of Seconds

Minutes

Visited

26

Testing session

• Preparation • The session • After the session

• Example – R2D2 project

27

Starten av dagen: • Sjekke at alt er på plass i laben; gavekort, utstyr, godteri, brus, batterier, pen og papir osv. • Skrive ut eksperimentet material som skal deles til deltagerne • Skrive ut tlf.nr. til de som kommer (mail fra Nordstat) • Informere resepsjonen at vi venter deltakere. Hvis deltaker kommer veldig tidlig, be resepsjonisten

henvise dem til sofa i resepsjonen. • Logge seg på PC og teste alt virker (ingen updates skal dukke opp i løpet av eksperimentet) • Gå inn på facebook og twitter og sjekk at ”thomtor” er logget ut • Heng IKKE FORSTYR på døra • Ha alltid mobilen med deg Eksperimentet (før deltakeren kommer) • Start SINTEF verktøyet (r2d2exp.jar). Det kan ta litt tid – IKKE KLIKK FLERE GANGER! • Start opp camtasia studio (dobbelt klikk) • Start opp uLog (høyre klikk på ikonet i ”system tray” og start) • Eksperiment (hent deltaker) • Ta med deltageren ned; hilse; introduksjon • Konfidensialitet skjema signeres • Husk å sette mobilen på vibrasjon • Start opptak i camtasia studio (høyreklikk, klikk ‘record the screen’, klikk record button) • …

28

29

Making sense of the data – Write up the results and help influence the design of

the specific interface – Presentation to developers and managers – Report should

• Include all flaws • Priorities • For each flaw: describe the problem, present the data,

priority, suggest a fix, estimate time/efforts for the fix – Report structure

• How you did usability testing and how you prepared • What happened during the testing • The implications and recomendations

30

Questions – Part I Go to the alarm app on you phone

– Difference between a summative and a formative usability test of this app

– Should we first test it with users or with experts? What do you think? Why?

– What are the differences between a usability test and an experiment with this app?

31

Questions – Part II Alarm app usability test • What is important to

remind participants before the usability test?

• What is the "think aloud" protocol? Should it be used in summative or formative studies?

• Properties of good tasks?

32