interpretive evaluation agenda simple user modeling interpretive evaluation - motivation - methods...
TRANSCRIPT
Interpretive Evaluation
Agenda• Simple user modeling• Interpretive Evaluation
- Motivation- Methods
Ethnography
• Fitt’s Law
Fall 2002 CS/PSY 6750 2
Simpler User Modeling
• How do attributes of users influence the design of user interfaces?
• Are there some design guidelines that we can derive from different attributes?
Fall 2002 CS/PSY 6750 3
User Profiles
• Attributes: attitude, motivation, reading level,
typing skill, education, system experience, task experience, computer literacy, frequency of use, training, color-blindness, handedness, gender,…
• Novice, intermediate, expert
Fall 2002 CS/PSY 6750 4
Motivation
• User Low motivation,
discretionary use Low motivation,
mandatory High motivation,
due to fear High motivation,
due to interest
• Design goal Ease of learning
Control, power
Ease of learning, robustness, control
Power, ease of use
Fall 2002 CS/PSY 6750 5
Knowledge & Experience
• Experience• task system
low low
high high
low high
high low
• Design goals Many syntactic and
semantic prompts Efficient commands,
concise syntax Semantic help
facilities
Lots of syntactic prompting
Fall 2002 CS/PSY 6750 6
Job & Task Implications
• Frequency of use High - Ease of use Low - Ease of learning & remembering
• Task implications High - Ease of use Low - Ease of learning
• System use Mandatory - Ease of using Discretionary - Ease of learning
Fall 2002 CS/PSY 6750 7
Evaluation
• Predictive Discount Usability (Mon) Cognitive Modeling-GOMS family (Wed) Cognitive Modeling-contextual (next
Mon)
• Interpretive (today)• User testing (later)
Fall 2002 CS/PSY 6750 8
Approaches
• Experimental (Lab studies, quantitative) Typically in a closed, lab setting
Manipulate independent variables to see effect on dependent variables
• Naturalistic (Field studies, qualitative) Observation occurs in “real life” setting
Watch process over time “Ecologically valid”
Fall 2002 CS/PSY 6750 9
Interpretive Evaluation
• Experimental: Formal and objective• Interpretive: More subjective
Concerned with humans, so no objective reality
Sociological, anthropological approach
• Users involved, as opposed to predictive approaches
Fall 2002 CS/PSY 6750 10
Beliefs
• Sees limitations in scientific hypothesis testing in closed environment Lab is not real world Can’t control all variables Context is neglected Artificial, short tasks
Fall 2002 CS/PSY 6750 11
IE Methods
• A number of different methods or techniques in this area exist Ethnography
Our main focus Contextual inquiry
More specific form of ethnography with a focus on asking questions
Field studyCommon notion, often equated with ethnography
Observational studyMuch more informal, just watching users
…
Fall 2002 CS/PSY 6750 12
Ethnography
• Deeply contextual study• Immerse oneself in situation you want
to learn about (anthropological and sociological roots) Observing people in their cultural context
• Interpretation of data is primary• Behavior is meaningful only in context
Fall 2002 CS/PSY 6750 13
Philosophy
• Argues that formal environment of controlled study is artificial --- Experimenter wields “power” over subject
• So … get into working environment of user
• -> Interpretation is primary, rather than data
Fall 2002 CS/PSY 6750 14
Objectives
• Understanding the user Understand goals and values Understand individual’s or group’s
interactions within a culture Try to make tacit domain knowledge explicit Do this in an unbiased fashion For UI designers: Improve system by finding
problems in way it is currently being used
Fall 2002 CS/PSY 6750 15
Techniques
• In person observation• Audio/video recording• Interviews
• “Wallow in the data”
Fall 2002 CS/PSY 6750 16
Observation is Key
• Carefully observe everything about users and their environment
• Think of describing it to someone who has never seen this activity before
• What users say is important, but also non-verbal details
Fall 2002 CS/PSY 6750 17
Observations
• Things of interest to evaluator Structure and language used in work Individual and group actions Culture affecting work Explicit and implicit aspects of work
• Example: Office work environment Business practices, rooms, artifacts, work
standards, relationships between workers, managers, …
Fall 2002 CS/PSY 6750 18
Interviews Important
• Have a question plan, but keep interview open to different directions
• Be specific• Create interpretations together with
users• At end, query “What should I have
asked?”• Record interviews
Fall 2002 CS/PSY 6750 19
Steps
• 1. Preparation Understand organization policies and work culture Familiarize yourself with system and its history Set initial goals and prepare questions Gain access and permission to observe & interview
• 2. Field study Establish rapport with users Observe/interview users in workplace and collect
all different forms of data Follow any leads that emerge from visits Record the visits
Rose et al ‘95
Fall 2002 CS/PSY 6750 20
Steps
• 3. Analysis Compile collected data in numerical, textual
and multimedia databases Quantify data and compile statistics Reduce and interpret data Refine goals and process used
• 4. Reporting Consider multiple audiences and goals Prepare a report and present findings
Fall 2002 CS/PSY 6750 21
One Technique
• Affinity Diagram• Process
Write down each quote/observation on a slip of paper
Put up on board Coalesce items that have affinity
If they are saying similar things about an issue
Give names to different groups (colors too) Continue grouping subgroups A hierarchy will be formed
Fall 2002 CS/PSY 6750 22
Why Useful?
• Can help designer gain a rich and true assessment of user needs Help to define requirements
• Uncovers true nature of user’s job Discovers things that are outside of job description or
documentation
• Allows you to play role of end-user Can sit in when real users not available
• Open-ended and unbiased nature promotes discovery Empirical study and task analysis are more formal –
ethnography may yield more unexpected revelations
Fall 2002 CS/PSY 6750 23
Types of Findings
• Can be both Qualitative
Observe trends, habits, patterns, …
QuantitativeHow often was something done, what
per cent of the time did something occur, how many different …
Fall 2002 CS/PSY 6750 24
Drawbacks
• Time required Can take weeks or months
• Scale Most use small numbers of participants just to
keep somewhat manageable
• Type of results Highly qualitative, may be difficult to present
and use
• Acquired skill Identifying and extracting “interesting” things is
challenging
Fall 2002 CS/PSY 6750 25
Ethnomethodology
• Concurrent/informed ethnography Study is being done in conjunction with
a system being developed + Helps keep focus on user throughout
design - Requires lots of time and coordination
Fall 2002 CS/PSY 6750 26
Cooperative Evaluation
• User is viewed as collaborator in evaluation, not a subject “Friendly approach”
• Relaxed version of think-aloud Evaluator and participant can ask each
other questions
Fall 2002 CS/PSY 6750 27
CE Methods
• Seeks to detect errors early in a prototype
• Experimenter uses tasks, also talks to participant throughout, asks questions
Fall 2002 CS/PSY 6750 28
Low Level Models
• Fitt’s Law Models movement times for selection tasks
• Basic idea: Movement time for a well-rehearsed selection task Increases as the distance to the target
increases Decreases as the size of the target increases
Fall 2002 CS/PSY 6750 29
Components
• ID - Index of difficulty
ID = log2 ( 2A/W )
bits result
Both quantities are distancesso unitless result
width toleranceof target
distance to move
Fall 2002 CS/PSY 6750 30
Components
• MT - Movement time
MT = b * ID orMT = a + b*ID
MT is a linear function of ID
Fall 2002 CS/PSY 6750 31
Original Experiment
• 1-DA W
Fall 2002 CS/PSY 6750 32
Exact Equation
• Run empirical tests to determine a and b in MT = a + b*ID
• Will get different ones for different input devices and ways the device is used
MT
ID
Fall 2002 CS/PSY 6750 33
Common Equation
• MT = a + b log2 (A/W + 1)
• Provides useful numbers
Fall 2002 CS/PSY 6750 34
Questions
• What do you do in 2D?
• Where can this be applied in user interface design?