Ch 14. Testing & modeling users

Steven PautzLauren Sullivan Jessica HerronChris Moore

The aims Describe how to do user testing. Discuss the differences between user testing, usability

testing and research experiments. Discuss the role of user testing in usability testing. Discuss how to design simple experiments. Describe GOMS, the keystroke level model, Fitts’ law and

discuss when these techniques are useful. Describe how to do a keystroke level analysis.

User Testing

• Developers test whether product is usable by the intended user population

• Similar to experimentation, but different from research• Aim is to improve existing design, rather than to

discover new knowledge

• Not designed to be replicable

• Often not published

Testing Medlineplus

• User testing was done to evaluate changes made after heuristic analysis (chapter 13)

• Goal of study was to identity usability problems in the revised interface• Categorization of information

• Users’ navigation strategies

Medlineplus: Testing Setup

• 9 participants from washington, DC area• 7 were female, all had some web experience• 5 tasks were developed, from frequently-asked

questions by web visitors• 5 scripts were created, one for each stage of the test• Testing was performed in a laboratory setting

Medlineplus:testing

• Every participant worked through the 5 tasks• Their progress through the site and “thinking

aloud” comments were recorded• After all tasks were completed, a post-test

questionnaire was given, and participants were asked their opinions

• Many different data items were collected

Medlineplus: Data Analysis

• Data was analyzed with a focus on:• Website organization

• Browsing efficiency

• Search features

• Conclusions showed several areas for improvement, which were given to the developers through a presentation and report

Medlineplus: Pros and Cons

Pros:

• Appropriate size, qualities of user group

• Adequate briefing material was created

• Informed consent form

Cons:

• Focused heavily on some user qualities (location, etc) at the expense of others

• Not much variance among tasks tested

Doing User Testing

• Central idea: controlling the test conditions.– Careful planning is necessary!

• Use DECIDE framework for successful study.

Determine the goals & Explore the questions

• User testing is most suitable for testing prototypes.• Focus the study:

– “Can users complete a certain task within a certain time, or find a particular item, etc.”

Choose a paradigm

• Usability testing paradigm• Involves:

– Recording data by combination of video and interaction logging.

– User satisfaction questionnaires.

– Interviews.

Identifying the practical issues: Design typical tasks

• Quantitative measures are obtained.• Types of data produced:

– Time to complete a task.– Time to complete a task after a specified time away

from the product.– Number and type of errors per task.– Number of errors per unit of time.– Number of navigations to online help or manuals.– Number of users making a particular error.– Number of users completing a task successfully.

Usability Engineering Specifications

• Current level of performance.• Minimum acceptable level of performance.• Target level of performance.

Identifying practical issues: Select typical users

• Find the ‘typical user’.• Most important characteristic:

– Previous experience with similar systems.

• Ex. Hutchworld – targeted towards cancer patients.

Identifying practical issues: Prepare the testing conditions

• Testing conditions include:– Controlled environment.

– Observation room.

– Viewing room.

– Reception area.

Deal with ethical issues

• Consent form.• Point out presence of

– One way mirrors

– Video cameras

– Use of interaction logging.

Evaluate, analyze, and present the data

• Performance measures are recorded from video and transaction logs.

• User tests involve a small number of participants.Basic descriptive statistics enable the evaluators to compare performance on different prototypes or systems.

Experimental Conditions

• In order to test a hypothesis, the experimenter has to set up the experimental conditions and find ways to control other variables that could influence the test result.– Condition 1: read screen of Helvetica– Condition 2: read screen of Times New Roman– Control Condition: read both on printed paper

Experimental Designs• Different participants - single group of

participants is allocated randomly to the experimental conditions.

• Same participants - all participants appear in both conditions.

• Matched participants - participants are matched in pairs, e.g., based on expertise, gender.

Advantages & disadvantagesParticipants Design

Advantages Disadvantages

Different No order effects Many subjects & individual differences could be a problem

Same Few individuals, no individual differences

Counter-balancing needed because of ordering effects. eg: Task A then B and vice versa.

Matched Same as different participants but individual differences reduced

Cannot be sure of perfect matching on all differences. Other variables could be having an effect.

Other Issues

• You need to determine where the experiment will take place as well as how the equipment will be set up.

• You also need to determine how the participants will be introduced to the study and what scripts will be needed to standardize the procedure.

• Pilot studies are useful to aid in this process

Menu Structure

• Study was designed to determine whether breadth is preferably to depth in organizing navigable menus.

• 3 Experimental Conditions: (top level, sub level, content level)

– 8 x 8 x 8 (slowest)– 16 x 32 (fastest)– 32 x 16 (middle)

• 19 experienced web users were given 8 random and unique searches for each condition.

• Breadth IS preferable to Depth

Predictive Models• Provide a way of evaluating products or designs

without directly involving users• Psychological models of users are used to test

designs• Less expensive than user testing• Usefulness limited to systems with predictable tasks

- e.g., telephone answering systems, mobiles, etc.• Based on expert behavior• GOMS is the most well-known predictive modeling

technique in human-computer interaction along with its “daughter”, the keystroke level model

GOMS• Goals - the state the user wants to achieve e.g., find a

website• Operators - the cognitive processes & physical

actions performed to attain those goals, e.g., decide which search engine to use

• Methods - the procedures for accomplishing the goals, e.g., drag mouse over field, type in keywords, press the go button

• Selection rules - determine which method to select when there is more than one available

Keystroke Level ModelGOMS has also been developed further into a quantitative model - the keystroke level model.

This model allows predictions to be made about how long it takes an expert user to perform a task.

Response times for keystroke level operators

Operator Description Time (sec)K Pressing a single key or button

Average skilled typist (55 wpm)Average non-skilled typist (40 wpm)Pressing shift or control keyTypist unfamiliar with the keyboard

0.220.280.081.20

P

P1

Pointing with a mouse or other device on adisplay to select an object.This value is derived from Fitts’ Law which isdiscussed below.Clicking the mouse or similar device

0.40

0.20H Bring ‘home’ hands on the keyboard or other

device0.40

M Mentally prepare/respond 1.35R(t) The response time is counted only if it causes

the user to wait.t

Benefits and Limitations

• Benefits– Allows comparative analyses for different interfaces or

computer systems• Example: Project Ernestine

– Viewed 2 systems to determine which would be better– Findings showed that a new system would slow down production

• Limitations– Not good for evaluations

• Does not allow for errors to be modeled, only expert performance

• Does not reflect multi-tasking (non-sequential procedures)

Fitts’ Law (Paul Fitts 1954)

• The law predicts that the time to point at an object using a device is a function of the distance from the target object & the object’s size.

• The further away & the smaller the object, the longer the time to locate it and point.

• Useful for evaluating systems for which the time to locate an object is important such as handheld devices like mobile phones

Summary & Key Points User testing is a central part of usability testing Testing is done in controlled conditions User testing is an adapted form of experimentation Experiments aim to test hypotheses by manipulating certain

variables while keeping others constant The experimenter controls the independent variable(s) but not the

dependent variable(s) There are three types of experimental design: different-

participants, same- participants, & matched participants GOMS, Keystroke level model, & Fitts’ Law predict expert,

error-free performance Predictive models are used to evaluate systems with predictable

tasks such as telephones