EPFL, spring 2012 - week 12!UX evaluation

today’s schedule

➝  review of UX evaluation plans ➝  test run of scripts ➝  individual walk throughs

overview

➝  about UX evaluations ➝ UX evaluation criteria ➝ UX evaluation methods ➝  tips and hints ➝  for more information

✱Attribution: parts of this presentation are based on "Forum Nokia’s UX Evaluation series

why UX evaluation? ➝  UX is a critical success factor for products and services

➝  We need confirmation during design and development that we are on the right track

➝  It is challenging for the software developer to keep an objective view of the developed software ➝  It is human nature to become “blind” of your own

outcomes ➝  Thus it is important to get a second opinion of the UX,

from outside of the development team

➝  Preferably from real users trying it in a realistic environment

The designer is not the user

Success factor

Simulate real use situations

Functionality Reliability Efficiency Usability Portability Maintainability

Suitability Maturity Time behavior Operability Installability Testability

Accuracy Fault tolerance Resource utilization

Learnability Adaptability Stability

… … ... … … …

usability and product quality

➝  there are many metrics to measure software performance and quality

Code coverage

Source lines of code

Function point analysis

Number of sw bugs

Implemented functional

requirements

SW Product Quality (ISO-9126)

UX and product quality

User Experience

Expectations, motives, actions, interpretations,

…

➝  measuring UX is not straight-forward

➝  there are many intervening issues

➝  product quality has a strong impact on UX

In this definition, UX is not a part of product quality.

A SW engineering

point-of-view?

Functionality Reliability Efficiency Usability Portability Maintainability

Suitability Maturity Time behavior Operability Installability Testability

Accuracy Fault tolerance Resource utilization

Learnability Adaptability Stability

… … ... … … …

SW Product Quality (ISO-9126)

UX evaluation criteria 1/5

basic usability heuristic UX drivers derived from the UX elements

as such issues

detailed for your product

drivers tailored for your product

➝  Nielsen (1993) on usability ➝  learnability

➝  Novices’ ability to reach a reasonable level of performance rapidly

➝  efficiency ➝  Expert user’s level of performance

➝  errors ➝  Number of errors users make ➝  User’s ability to recover from

errors ➝  Existence of catastrophic errors?

➝  memorability ➝  Casual user’s ability to remember

how to use a system

➝  satisfaction ➝  User’s subjective assessment of

the system

useful ness

utility

relia bility

ease-of-use

effici ency

acces sibility

connec ting

people

identifi cation

pleasure stimulation

usabi lity

social value

enjoy ment

UX

UX evaluation criteria 2/5

➝  Evaluate the design against the specified UX artefacts

use case storyboard accessing phone from web browser

Melvin, 35!

•  Engineer from Germany •  Lives with his wife, 2 kids and a dog. •  Both parents need to sometimes travel due their jobs, and kids have lots of hobbies. •  Would like better manage his everyday schedules with family.

Jason, 25!

•  Customer Service Assistant. •  Keeps a blog for his friends and family Uses his device for capturing images to blog. Most of the images related to snowboarding. •  Plays guitar in a wildly not-yet-so-popular band.

Louise, 27!

•  Web designer, graduate student on university. •  SMS addict - 50 messages every day. •  Likes arts, especially indie movies and old films. •  Is currently organizing surprise parties for her friend.

UX evaluation criteria 3/5

➝  evaluate against benchmark ➝  products and experiences

UX evaluation criteria 4/5

➝  Run your tests with all the target devices and form factors

Bar Slide

Clamshell Swivel

Touch screen Touch only screen

Full hw qwerty Full sw qwerty

Resistive screen Capacitative screen

UX evaluation criteria 5/5

➝  the ultimate criterion: ➝  potential users try the product in their mobile context

people places Things time

context

Direct, spoken criteria

Indirect, unspoken criteria - originating from the

context

planning the UX evaluations 1/2

➝  for any UX artifacts, you can think can and should it be evaluated, somehow

evaluate concepts and designs

A step-by-step process would

be easy to manage.

In practice, design and

evaluation are often in a parallel

and iterative relationship.

specify needs

and context of use define the UX

concept create UI designs

specify needs and context of

use define the UX

concept create UI designs

evaluate the UI designs

planning the UX evaluations 2/2

Evaluate concepts and designs

➝  What artefacts do we want to evaluate ➝  Sketches, visualisations, prototypes,...

➝  What issues are we evaluating? ➝  Concept, interaction, grahical design, ...

➝  Who will do the evaluation? ➝  Experts, end users,...

My cool travel- mate concept

UX evalution: expert evaluation ➝  Usability experts evaluate a system based on

common heuristics (i.e. design principles) and personal expertise ➝  Common heuristics are based on knowledge gained

through expertise and experience ➝  Knowledge about perceptional and cognitive processes

- such as the function of memory – are utilized ➝  At least two experts take part in the evaluation to

ensure reliable results ➝  At the beginning of an expert evaluation, the value of

the system to its user and buyer/provider is determined ➝  Based on these values, usability criteria are defined

➝  At the end, the findings documented and prioritized. Solutions proposals are made

➝  Also user tasks can be defined and walkthroughs conducted as part of the evaluation

UX evaluation: focus groups ➝  a group of people (4-7) that has

something in common will be invited to semi-structured group discussion session to share their views on certain topic. Lasts usually 2 hours

➝  face-to-face sessions ➝  Sometimes on-line

➝  variations of the method ➝  Exploratory ➝  Feature prioritization ➝  Competitive analysis ➝  Trend explanation

UX evaluation: usability testing ➝  the test user is asked to conduct certain tasks with the

user interface under design ➝  paper prototypes can be used ➝  users are asked to think aloud

➝  sometimes good to test the UI with pairs of users to

trigger more discussion or to test social aspects ➝  usability problems are identified and prioritized based on

the user data gathered during the usability tests

Moderator

Test user Test cases

Observer

Tested sw

Special equipment can be useful to catch the

actions on mobile devices

UX evaluation: observation ➝  the researcher does not interrupt the user

but just observes the user’s actions ➝  the user is followed usually shorter time

than when shadowing in her/his own environment, and usually only certain activities/time period are observed

➝  good for studying users that cannot be interrupted

➝  reveals task flows and possible detectable problems in them and behavioral patterns but not motivations, attitudes, values, concerns

If observation is not possible (e.g. in hospital) video

recording can be alternative. Video can be gone through

with the participant(s) afterwards

don't listen to users

➝ watch users attempting to perform tasks in the UI ➝ watch what people actually do ➝ be careful with what people say they do ➝ don't believe what people predict they may do in the

future.

when you listen

beware: ➝  users often tell you what they think you want to

hear ➝ often based only on recall ➝  rationalized behavior

UX evaluation: make a mix

➝  an expert evaluation and a usability test can be used as complementary methods

➝  an expert evaluation require less time and preparation than a usability test

➝  expert evaluations are recommended before usability tests in order to support designing the test tasks

➝  a satisfaction questionnaire can be combined with a usability test

UX evaluation: tips and hints ➝  the UX evaluation criteria should be in-line with

the defined UX drivers, targets and requirements ➝  make sure to share your UX evaluation criteria

with the designers before they start their work ➝  do evaluations with real users

➝  expert evaluations can provide quick useful findings ➝  try to have a end user mindset

➝ ideally your evaluation methods should scale to weekly (or even daily) evaluations ➝  design & evaluation iteration loops do not have to be

big monolithic steps

➝  UX evaluation(s) should be an integral part of software development process, just like software testing

design evaluate

iterate