evaluation of adaptive web sites 3954 doctoral seminar 1 evaluation of adaptive web sites elizabeth...

Evaluation of Adaptive Web Sites3954 Doctoral Seminar 1

Evaluation of Adaptive Web Sites

Elizabeth LaRue

by


Presentation Resources Gena, C. (2005). Evaluation of Adaptive Web Sites.

Retrieved September 6, 2005, from Courseweb.

Gena, C. (2002, January 13-16). An empirical evaluation of an adaptive web site. Paper presented at the International Conference on Intelligent User Interfaces, San Francisco, CA.

Marucci, L., & Paterno, F. (2002). Design and evaluation of an adaptive virtual guide for web applications. UAIS: Universal Access in the Information Society, 1(3), 163-176.

Cheverst, K. Davies, N. Mitchell, K. et al. (1-6 April 2000). Developing a context-aware electronic tourist guide: some issues and experiences. CHI. [electronic source] retrieved from ACM Digital Library.


Objectives

• Review associated terminology

• Describe project phases

• Examine various evaluation methods


Terminology

• Adaptive Web Sites – present and format information material automatically on a web page for the user

• Evaluation – to examine carefully 1

• Empirical – relying on or gained from observation or experiment rather than theory 1

1 Houghton Mifflin Company. (1996). Webster's ii new riverside dictionary (Revised Edition ed.). Boston: Houghton Mifflin Company.


Benefits of Evaluation

• Helps design

• Insight into user behavior

• Tests usability and functionality

• Assists with user models


Adaptive Systems Evaluation

• Consider a layered approach – separate evaluation into the adaptive components of the adaptive systems

A. Content layer

B. Interface layer


Software Lifecycle DevelopmentWaterfall Methodology

Planning/Requirements

Design/Analysis

Code/Program

Testing

Implementation

Evaluation

Evaluation

Evaluation

Evaluation

Evaluation

(development)


Planning/Requirements

Why?

Continual EVALUATION helps to complete the requirements specification document

Assists in Focusing on users – typical behavior, actions, needs, etc… Focusing on tasks – actions, needs, etc…


Why?

Design/Analysis

Assists in Prohibiting expensive design mistakesProhibiting costly redesign


Code/Program (development)

Why?

Assists in Evaluating overall quality of a system


Before Evaluating

• Data collection = query technique

– Usually used at each step in the life cycle– Usually cheap– Usually simple


• Questionnaires

• Interviews

Before Evaluating - Tools


Verbal and think aloud (pre and post)

User’s observation

Logging Use

Before Evaluating – ToolsObservational Methods


• Formal

• Analytical

• Empirical

Three Types of Evaluation


• Rarely used

• Look at prediction of performance, complexity and learnability

Formal Type of Evaluation


Formal Methods

• Task Analysis

• Cognitive – GOMS– KLM (Keystroke Level Model)

• Socio-technical models


• Used without users, must have clear evaluation criteria of the system

• Benefits– Quick– Cheap– Reasonably effective

Analytical Type of Evaluation


Analytical Models

• Heuristic Evaluation

• Cognitive walkthrough

• Accessibility

• Automatic Usability Testing and Web usage mining


Empirical Type of Evaluation

• Best for objective assessment of design• Best for broadest range of usability problems

Requirements to do empirical evaluation

1. Careful planning

2. Careful execution

3. Users must represent actual user population

4. Must have expense account – pay users


Empirical Methods• Contextual evaluation• Focus group• Systematic Observation• Expert review• Wizard of Oz prototyping• Prototyping• Card sorting• Cooperative evaluation• Participative evaluation• Usability testing• Accessibility• Automatic Usability Testing and Web usage mining• Controlled experiments• Ethnography• Grounded theory• Metric to evaluate the effectiveness of adaptive contents


Specific Examples


Mixed Methods – Empirical Methodsurveyparallel designtask analysisquestionnaire - post



Evaluation of adaptive commercial web site



Test: 1. Measure the task completion time 2. Amount of within page navigation 3. The satisfaction of the users

Evaluation MethodsInteraction simulation with pre-testAccomplish task in non-adaptive version and then repeat in adaptive version

Study Sample14 subjectsAged 24-35High Internet knowledge and web browsers and use the Internet during work



Specific Example 2

Marucci, L., & Paterno, F. (2002). Design and evaluation of an adaptive virtual guide for web applications. UAIS: Universal Access in the Information Society, 1(3), 163-176.

Mixed Methods – Observational Method

Questionnaire

Think aloud


Evaluation of adaptive museum web site

Marucci, L., & Paterno, F. (2002). Design and evaluation of an adaptive virtual guide for web applications.

UAIS: Universal Access in the Information Society, 1(3), 163-176.

http://giove.cnuce.cnr.it/servlets/StartVisitEng



Test: 1. Does the virtual guide make a visit more pleasant and instructive

2. Learn if the type of information provided by the guide was interesting to the user

3. Is the presentation of the material effective or could be improved

Evaluation MethodsThink aloud without tasks – minimal 30 minutes with and without adaptive systemPost-questionnaire – 29 questions

Study Sample40 subjectsAged 20-53 (16 males and 24 females)Previous web experience and half rarely accessed a museum web site







Think aloud methodology

• Evaluators recorded particularly negative and positive reactions• Evaluators noted specific paths followed by users• Evaluators were careful to not make suggestions to users

Questionnaire• 15 questions on general usability of interface, navigation and ease of access• 5 questions related to each of the five types of information provided by the

virtual guide• 9 questions related to user personal information, i.e. kn of museum application

domains




Specific Example 2A

Evaluation of tourist web site: (GUIDE)

Mixed MethodsExpert walkthrough – one hourTalk aloudPost-interview

Study Sample4 expert subjectsKnowledge area of user-centered design and computer supported learning




http://www.guide.lancs.ac.uk/whatisguide.html



Prototype Evaluation Methods Talk aloud – without tasks Time stamped log files Post semi-structured interview

Study Sample60 volunteer subjectsAged 10-70 Knowledge area of user-centered design and computer supported learning



Thank You

evaluation of adaptive web sites 3954 doctoral seminar 1 evaluation of adaptive web sites elizabeth...

Documents

web page

web applications

adaptive components

adaptive systemsa

clear evaluation criteria

adaptive virtual guide

user behaviortests usability

user modelsadaptive