evaluation paradigms & techniques is 588 spring 2008 dr. d. bilal
TRANSCRIPT
![Page 1: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/1.jpg)
Evaluation Paradigms & Techniques
IS 588
Spring 2008
Dr. D. Bilal
![Page 2: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/2.jpg)
Overview
• Evaluation is performed to determine how well a certain product design meets user needs.
• Need to decide what to evaluate?– Guided by goals, theory, model, etc.
• What to evaluate determines how to do the evaluation
![Page 3: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/3.jpg)
Evaluation Paradigms
• Quick & Dirty
• Usability testing
• Field studies
• Predictive evaluation
![Page 4: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/4.jpg)
Quick & Dirty
• Informal
• Designers or evaluators meet informally with users– Gather information about product design– Gather suggestions for design improvements
• Inexpensive
• Not time consuming
![Page 5: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/5.jpg)
Usability Testing
• Formal assessment• Measures user performance on predefined tasks
– Tasks structured based on purpose of evaluation
• Controlled by evaluator• Performance observed and/or captured
– EXAMPLES?– based on questions guiding usability testing (i.e., what
the evaluator wants to find)
![Page 6: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/6.jpg)
Usability Testing
• Typically quantitative
• Interviews and questionnaires can result in qualitative assessments – User comments, quotes of likes/dislikes, etc.
• A mix method is ideal– WHY?
![Page 7: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/7.jpg)
Usability Testing
• Not performed in a naturalistic setting
• Activities can be captured/recorded using software (e.g., Morae, HyperCam, Camtasia), or videotape
• Evaluator may take observational notes while activities being captured
![Page 8: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/8.jpg)
Field Studies
• Naturalistic setting – User interacts with system as part of a daily
routine– No tasks given by evaluator– Evaluator observes and records activities, OR
uses software to capture activities, OR…
• Can be qualitative and quantitative– HOW?
![Page 9: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/9.jpg)
Predictive Evaluation
• Experts place themselves in the users’ shoes to predict usability problems
• Guided by heuristics– Quick, inexpensive– Limitations
• WHAT ARE THEY?
![Page 10: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/10.jpg)
Evaluation Techniques
• Observe user
• Gather user opinion
• Gather expert opinion
• Test user performance
• Model user performance
• Mix method (2 or more techniques)
![Page 11: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/11.jpg)
DECIDE Framework
• Determine goals
• Explore/set questions to be answered
• Choose suitable paradigms and techniques
• Identify issues (e.g., how to recruit participants)
![Page 12: Evaluation Paradigms & Techniques IS 588 Spring 2008 Dr. D. Bilal](https://reader036.vdocuments.net/reader036/viewer/2022082711/56649ead5503460f94bb3877/html5/thumbnails/12.jpg)
DECIDE Framework
• Decide on tackling ethical concerns (e.g., use of human subjects, privacy)
• Evaluate, interpret, present data