eindhoven technische universiteit measuring user satisfaction through experiments b. de vries

EindhovenTechnische Universiteit

Measuring User Satisfactionthrough Experiments

B. de Vries


Domotica

Computable: 5 November 2004

Wildgroei

Toekomstdroom

Losse deelmarkten


Innovatie

• Juiste product ?

• Juiste doelgroep ?

• Juiste distributie ?

• Juiste tijd ?

• Juiste marketing ?

• …


Evaluation

• Observational > Case studies

• Experimental > Research


Characteristics

Empirical: Gather evidence through observation and measurement that can be replicated by others

• Measurement

• Replicability

• Objectivity


Variables

• Independent: Cause

• Dependent: Effect


Scientific research

• Validity: Are you measuring what you claim to measure( measuring the right thing)

• Reliability: The ability to produce the same results under the same condition(Measuring things right)

• Error: The difference between our measurements and the value of the construct we are measuring


Validity

Internal validity problems• Group threats, regression to the mean,

time threats, history, maturation, instrumental change, differential mortality, reactive and experimenter effects

External validity problem• Over-use of special participants group,

restricted number of participants


Between groups

Treatment(experimental gp.)

No Treatment(control gp.)

Measurement

Measurement

Randomallocation


Measuring User satisfaction

• Virtual Reality

• Bayesian Belief Networks

Desk-Cave


Set-UP

• 2 synchronized PC’s with dual monitor output

• 4 LCD Projectors


Features

• 1 : 1 Scale

• 3DS import

• Immersion

• Interaction


Bayes Theorem

)(

)()|()|(

BP

APABPBAP

From: Evaluation and Decision (7M834)


Bayesian Belief Network

Norman

Late

MartinLate

TrainStrike


Node Probability Table

Norman

Late

MartinLate

TrainStrike

Train Strike

Norman late

True False

True 0.8 0.1

False 0.2 0.9


NPT’s

Train Strike

Martin late True False

True 0.6 0.5

False 0.4 0.5

Train strike

True 0.1

False 0.9


Analyzing a BBN

p(Norman late) = p(Norman late | train strike) * p(train strike) + p(Norman late | no train strike) * p(no train strike)= (0.8 * 0.1) + (0.1 * 0.9) = 0.17

Marginal probability

Conditional probability

p(Train strike|Norman late) = ( p(Norman late|train strike) * p(train strike) ) / P(Norman late) = (0.8 * 0.1) / 0.17 = 0.47


Measuring User Satisfaction Using Virtual Reality and Bayesian Belief Networks.

01.11.2004

Maciej A. Orzechowski


Motivations, aims

Current techniques for measuring user preferences (CA, MM, interview) are artificial, lengthy or expensive.

For good results we need to get the respondents more involved in the measurement.

Can Virtual Reality (VR) improve the quality of measuring preferences: more involved and higher reliability?

The aim of this project was to develop and test an interactive VR tool for measuring housing preferences.


VR System

MuseV3 – a Virtual Reality application with functionality of a simple CAD system.

Two categories of modifications:

Structural modifications (change layout).

Textural modifications (change visual impression).


Structural Modifications

Change of internal and external dwelling’s layout.

The most important for estimating user preferences.

Include following commands: create/resize space; insert openings.

Direct impact on overall costs of the dwelling.


MuseV3 in Desktop CAVE


Bayesian Belief Network

Non-obtrusive interactive method to collect housing preferences.

Potential advantages

Interaction with the model during the time of preferences estimation.

Incremental learning.

Possibility to assess:

where the knowledge about preferences is most uncertain.

consistency of measurements.


Bayesian Belief Network cont.

A Bayesian Belief Network (BBN) captures believed relations (which may be uncertain, stochastic, or imprecise) between variables, which are relevant to some problem.

Lounge Ext(β1)

Garage Ext(β2)

Extra Kitchen

(β3)

2 Bedrooms(β4)

First FloorExt (β5)

DormerWindow (β6)

Choice ofLounge Ext

Choice ofGarage Ext

Choice ofExtra

Kitchen

Choice of2 Bedrooms

Choice ofFirst Floor

Ext

Choice ofDormerWindow

Price (γ)

Family Situation Age


CPT calculation

set B, G CPT to uniformprobability

calculate utility for eachChoice state, according to

each combination of states ofnodes B, G (eq. 2)

based on utilities, calculateprobability for each Choice

state, for each combination ofnodes B, G (eq. 3)

For each Choice node

B G

Choice

B-CPT State 1 PB1 State 2 PB2 State 3 PB3

Choice-CPT Choice-State 1 Choice-State 2 B-State 1 G-State 1 P 11 P 12 B-State 2 G-State 1 P 21 P 22 B-State 3 G-State 1 P 31 P 32 B-State 1 G-State 2 P 41 P 42 … … … …

G-CPT State 1 PG1 State 2 PG2 State 3 PG3 State 4 PG4 State 5 PG5


Learning process

makedesignchoices

ultimatedesign

solution

update CPT's of nodes B, G

newrespondent

Y

N

set choiceto selected

state


Convergence


Utility Convergence


Experiment

1600 letters -> 100 answers -> 64 respondents.

Respondents were people searching for a house or who just bought one.

4 kinds of 2 types of tasks (2 traditional, 2 based on MuseV3):

CA: Verbal Description Only (VDO) Multimedia Presentation (MM).

BBN: Preset Options (PO) Free Modification (FM).

Each respondent completed both types of tasks.


Observed-Predicted


Conclusions

The results support the potential of the suggested approach.

The results suggests higher involvement of respondents.

This approach is non-obtrusive compared to different preference measurement techniques.

The system (tool) can be used to:

To assist individual users in creating their own design.

To derive market potential of housing designs at aggregate level.


Domotica Experiments

• Alarmering: inbraak, zorg, brand

• Autom. Verlichting

• Autom. Zonwering

• …

eindhoven technische universiteit measuring user satisfaction through experiments b. de vries

Documents

effect slide

vries slide

orzechowski slide

pnorman late train strike

decision 7m834 slide

deskcave slide

train strike true0

ptrain strike pnorman