understanding real life website adaptations by investigating the relations

23
Understanding Real-Life Website Adaptations by Investigating the Relations between User Behavior and User Experience Mark P. Graus Martijn C. Willemsen Kevin Swelsen

Upload: mark-graus

Post on 14-Aug-2015

100 views

Category:

Science


0 download

TRANSCRIPT

Page 1: Understanding real life website adaptations by investigating the relations

Understanding Real-Life

Website Adaptations by

Investigating the Relations

between User Behavior and

User Experience

Mark P. Graus

Martijn C. Willemsen

Kevin Swelsen

Page 2: Understanding real life website adaptations by investigating the relations
Page 3: Understanding real life website adaptations by investigating the relations

• Three countries

– Netherlands

– Belgium

– UK

• Aimed at IT/CE-enthusiasts

• Second Biggest IT website in the

Netherlands: 8+ mln

pageviews/month

• Editorial Board + Price Comparison

• 1.500 products tested and reviewed

each year

• Active Community

3

Hardware.Info Online

Page 4: Understanding real life website adaptations by investigating the relations

Can we provide visitors with better content based

on an educated guess of their interests?

4

Page 5: Understanding real life website adaptations by investigating the relations

5

Hardware.info

Page 6: Understanding real life website adaptations by investigating the relations

Introducing a User-Centric Evaluation Framework

6

Page 7: Understanding real life website adaptations by investigating the relations

7

User-Centric Evaluation Framework

Predicted Segment

Shown Content

PerceivedAccuracy

Perceived Effectiveness

NavigationBehavior

Knijnenburg, B. et al. 2012. Explaining the user experience of recommender systems. UMUAI.

Page 8: Understanding real life website adaptations by investigating the relations

• When looking at what behavior is relevant, intuition might point us in

the wrong direction

Number of videos watched in a video browsing system correlated with

lower system satisfaction.

Knijnenburg et al., 2010, Receiving recommendations and providing

feedback: The user-experience of a recommender system

8

Why User Evaluation?

Page 9: Understanding real life website adaptations by investigating the relations

• During 2 weeks on hardware.info we ran an online experiment

• Collected 2 weeks worth of data

– 100k unique visitors

– 3k completed surveys

9

Process

Enter5

PageviewsSidebar Element

Pageviews Survey Pageviews

Page 10: Understanding real life website adaptations by investigating the relations

Analysis:

Predictive Modeling (Post-Hoc)

Evaluation of Adaptation

Evaluation of Adaptation10

Page 11: Understanding real life website adaptations by investigating the relations

Data

5 pageviews

Rest of VisitContentEUPHCMix

Survey Data

To what extent are you interested in HC/EUP?

11

Predictive Modeling

Model (Behavioral

Labels)

Labels (Behavioral Data)Labels

(Survey Data)

Model (Survey Labels)

Page 12: Understanding real life website adaptations by investigating the relations

12

Comparison of Predictions

Model (Survey Data)

Model (Behavioral Data)

PredictionsPredictions

Shown Content

PredictedSegment

Predicted Segment

Rest of Visit

HC HC HC • Clicks on Sidebar Element

EUP EUP EUP • Pageviews

Mix • Sessions

Explain

Page 13: Understanding real life website adaptations by investigating the relations

AIC

Labels Clicks on Sidebar Clicks on Sidebar (Boolean)

Pageviews Sessions

Behavior 834,821.3 26,910.6 23,362.0 517,453.3

Survey 832,555.5 26,832.5 23,270.2 514,761.0

13

Regression Model Fit

Page 14: Understanding real life website adaptations by investigating the relations

• A model based on Survey Data provides predictions that better

describe response to the Sidebar Element than models based on

Behavioral Data

• Despite less information (3k vs 100k)

• We are predicting segments for 100.000 visitors while using data

from only 3,000

14

Conclusion

Page 15: Understanding real life website adaptations by investigating the relations

Effects of the Adaptation

Evaluation of Adaptation15

Page 16: Understanding real life website adaptations by investigating the relations

16

Predicted Segment

Shown Content

(Congruency)

Perceived Accuracy

PerceivedEffectiveness

NavigationBehavior

Page 17: Understanding real life website adaptations by investigating the relations

17

Predicted Segment

Shown Segment

(Congruency)

Perceived Accuracy

PerceivedEffectiveness

NavigationBehavior

“The Items in the Sidebar matched my preferences.”

Page 18: Understanding real life website adaptations by investigating the relations

18

Predicted Segment

Shown Segment

(Congruency)

Perceived Accuracy

PerceivedEffectiveness

NavigationBehavior

“The sidebar helps me in finding new and interesting articles.”

Page 19: Understanding real life website adaptations by investigating the relations

19

Predicted Segment

Shown Segment

(Congruency)

Perceived Accuracy

PerceivedEffectiveness

NavigationBehavior

0

0.02

0.04

0.06

0.08

0.1

0.12

Non-congruent congruent

Content

Proportion of Visitors that Click Sidebar Element

HC

EUP

• Congruent Content Leads to • More Clicks on the Sidebar element• More Pageviews• More Visits

• Stronger effects for people interested in EUP

Page 20: Understanding real life website adaptations by investigating the relations

20

Putting it All Together: Path Model

Adaptation

ClickedINT

perceived

AccuracySSA

.212 (.09)

p<.05

.595 (.024)

p<.005

.103 (.03)

p<.005

-.138 (.056)

p<.05

perceived

EffectivenessEXP

HC Segment(versus EUP)

PC

Congruent(versus non-

congruent) OSA

HC:Congruent

OSA/PC

Page 21: Understanding real life website adaptations by investigating the relations

Take Home Message

21

Page 22: Understanding real life website adaptations by investigating the relations

• If you do Predictive Modeling of Latent User Characteristics

– Using a single question provides ground truth

• Reliable

• Cheap: 3,000 surveys cost us one SSD and one mobile

phone

• If you want to Ground Behavioral Changes in User Experience

– Use a full survey and analysis

• Answer the question: Why does behavior change?

22

Consider Surveys!

Page 23: Understanding real life website adaptations by investigating the relations

• Our thanks also goes out to

– hardware.info

– Bart Knijnenburg

• Questions/remarks?

Mark Graus – PhD Student

Human-Technology Interaction Group

[email protected]

https://twitter.com/newmarrk

https://linkedin.com/in/markgraus

http://www.marrk.nl

23

Thank You