understanding real life website adaptations by investigating the relations

Understanding Real-Life

Website Adaptations by

Investigating the Relations

between User Behavior and

User Experience

Mark P. Graus

Martijn C. Willemsen

Kevin Swelsen

• Three countries

– Netherlands

– Belgium

– UK

• Aimed at IT/CE-enthusiasts

• Second Biggest IT website in the

Netherlands: 8+ mln

pageviews/month

• Editorial Board + Price Comparison

• 1.500 products tested and reviewed

each year

• Active Community

3

Hardware.Info Online

Can we provide visitors with better content based

on an educated guess of their interests?

4

5

Hardware.info

Introducing a User-Centric Evaluation Framework

6

7

User-Centric Evaluation Framework

Predicted Segment

Shown Content

PerceivedAccuracy

Perceived Effectiveness

NavigationBehavior

Knijnenburg, B. et al. 2012. Explaining the user experience of recommender systems. UMUAI.

• When looking at what behavior is relevant, intuition might point us in

the wrong direction

Number of videos watched in a video browsing system correlated with

lower system satisfaction.

Knijnenburg et al., 2010, Receiving recommendations and providing

feedback: The user-experience of a recommender system

8

Why User Evaluation?

• During 2 weeks on hardware.info we ran an online experiment

• Collected 2 weeks worth of data

– 100k unique visitors

– 3k completed surveys

9

Process

Enter5

PageviewsSidebar Element

Pageviews Survey Pageviews

Analysis:

Predictive Modeling (Post-Hoc)

Evaluation of Adaptation

Evaluation of Adaptation10

Data

5 pageviews

Rest of VisitContentEUPHCMix

Survey Data

To what extent are you interested in HC/EUP?

11

Predictive Modeling

Model (Behavioral

Labels)

Labels (Behavioral Data)Labels

(Survey Data)

Model (Survey Labels)

12

Comparison of Predictions

Model (Survey Data)

Model (Behavioral Data)

PredictionsPredictions

Shown Content

PredictedSegment

Predicted Segment

Rest of Visit

HC HC HC • Clicks on Sidebar Element

EUP EUP EUP • Pageviews

Mix • Sessions

Explain

AIC

Labels Clicks on Sidebar Clicks on Sidebar (Boolean)

Pageviews Sessions

Behavior 834,821.3 26,910.6 23,362.0 517,453.3

Survey 832,555.5 26,832.5 23,270.2 514,761.0

13

Regression Model Fit

• A model based on Survey Data provides predictions that better

describe response to the Sidebar Element than models based on

Behavioral Data

• Despite less information (3k vs 100k)

• We are predicting segments for 100.000 visitors while using data

from only 3,000

14

Conclusion

Effects of the Adaptation

Evaluation of Adaptation15

16

Predicted Segment

Shown Content

(Congruency)

Perceived Accuracy

PerceivedEffectiveness

NavigationBehavior

17

Predicted Segment

Shown Segment

(Congruency)

Perceived Accuracy

PerceivedEffectiveness

NavigationBehavior

“The Items in the Sidebar matched my preferences.”

18

Predicted Segment

Shown Segment

(Congruency)

Perceived Accuracy

PerceivedEffectiveness

NavigationBehavior

“The sidebar helps me in finding new and interesting articles.”

19

Predicted Segment

Shown Segment

(Congruency)

Perceived Accuracy

PerceivedEffectiveness

NavigationBehavior

0

0.02

0.04

0.06

0.08

0.1

0.12

Non-congruent congruent

Content

Proportion of Visitors that Click Sidebar Element

HC

EUP

• Congruent Content Leads to • More Clicks on the Sidebar element• More Pageviews• More Visits

• Stronger effects for people interested in EUP

20

Putting it All Together: Path Model

Adaptation

ClickedINT

perceived

AccuracySSA

.212 (.09)

p<.05

.595 (.024)

p<.005

.103 (.03)

p<.005

-.138 (.056)

p<.05

perceived

EffectivenessEXP

HC Segment(versus EUP)

PC

Congruent(versus non-

congruent) OSA

HC:Congruent

OSA/PC

Take Home Message

21

• If you do Predictive Modeling of Latent User Characteristics

– Using a single question provides ground truth

• Reliable

• Cheap: 3,000 surveys cost us one SSD and one mobile

phone

• If you want to Ground Behavioral Changes in User Experience

– Use a full survey and analysis

• Answer the question: Why does behavior change?

22

Consider Surveys!

• Our thanks also goes out to

– hardware.info

– Bart Knijnenburg

• Questions/remarks?

Mark Graus – PhD Student

Human-Technology Interaction Group

m.p.graus@tue.nl

https://twitter.com/newmarrk

https://linkedin.com/in/markgraus

http://www.marrk.nl

23

Thank You