toward better interactions in recommender systems: cycling and serpentining approaches for top-n...
TRANSCRIPT
Toward Better Interactions in Recommender Systems:
Cycling and Serpentining Approaches for Top-N Item Lists
Qian Zhao, F. Maxwell Harper, Joseph A. Konstan (GroupLens)Gediminas Adomavicius (Dept. of Information and Decision
Sciences, University of Minnesota)Martijn Willemsen (Human-Technology Interaction group,
Eindhoven University of Technology)
1
Recommender Systems
» Recommender systems typically display the top-N recommended items in order.
2
Example: MovieLens.org Top Picks
3
What’s wrong with this recommender?
4
1st Visit
5
2nd Visit
6
3rd ... Visit
7
8
What’s wrong with this recommender?
9
1st Page
10
5th Page
11
10th … Page
12
13
Excitement
Show Time/Further Exploration
Re-thinking Top-N Recommendation Lists
» Static Top-N in order à the best design for user interaction and temporal experience
» Two missing factors• Fresh vs. stale• Further exploration à worse quality/experience
14
Prior Work: Dynamic/Interactive Recommenders
» Classics• Temporal dynamics (Koren et al. 2010)• CARS (Adomavicius et al. 2011)• Incremental matrix factorization (Luo et al. 2012)
» Interactive (reinforcement) machine learning• Markov decision processes (Shani et al. 2002)• Contextual bandits (Lu et al. 2010)
15
This Work: Cycling and Serpentining
» Cycling demotes items that have been viewed several (3+) times, exposing fresher recommendations.
» Serpentining spreads top recommended items across several pages, offering high-quality items on each page as a user continues to explore.
16
Cycling
Movie Score
M1 5.0
M2 4.9
M3 4.6
M4 4.0
17
Movie Score #display
M1 5.0 3
M2 4.9 3
Cycling
Movie Score
M1 5.0
M2 4.9
M3 4.6
M4 4.0
18
Movie Score #display
M3 4.6 0
M4 4.0 0
M1 5.0 3
M2 4.9 3
SerpentiningMovie Score
M1 5.0
M2 4.9
M3 4.8
M4 4.7
M5 4.6
M6 4.5
M7 4.4
M8 4.3
19
Movie Score
M1 5.0
M3 4.8
M5 4.6
M7 4.4
p. 1
p. 2
p. 1
p. 2
SerpentiningMovie Score
M1 5.0
M2 4.9
M3 4.8
M4 4.7
M5 4.6
M6 4.5
M7 4.4
M8 4.3
20
Movie Score
M1 5.0
M3 4.8
M5 4.6
M7 4.4
M2 4.9
M4 4.7
M6 4.5
M8 4.3
p. 1
p. 2
p. 1
p. 2
When to Cycle
» Within-session: each time when users go back to home page• More perceived change but more confusion?• Reflective of current recommenders in terms of the
change, e.g. Youtube.» Between-session: when users sign in next time• Less disorienting?• But, can users perceive the change?
21
The novel contribution of this work
» Cycling and serpentining approaches are not studied before.• e.g. Youtube is not cycling (Davidson et al. 2010).
» Better understand the effects of dynamic top-N lists on user experience.
22
A between-subjects field experiment on MovieLens.org» Measurements• Objective activity level• Subjective perception (through surveys)
23
Objective Activity Level
» opt out rate (bad experience)» number of page views (negative efficiency or
positive engagement)» number of interested actions. i.e. clicks,
wishlist (positive engagement)» interested rate: number of interested actions
per page view (positive efficiency)
24
Subjective Perception
» Specific aspects• accuracy, familiarity, diversity, novelty• change, freshness, confusion, boredom
» Overall experience• usefulness and satisfaction
» Surveys embedded in user browsing activities, i.e. sampling user experience
25
Survey Prompt
26
Survey Display
27
28
Data
» March 22 – May 14, 2016 (1.6 months)» 5158 invited users (having more than 15
ratings and two sessions)» 987 users joined the experiment» Analyzing activities of the first half month
(802 users) and all survey responses (~900)
29
Activity Statistics in the Experiment
Activity #Activities #Usersfront page 43,371 987
top picks page 43,231(mean page depth: 9.14) 821
rating 163,242 943click 107,955 924
wishlisting 23,032 473
30
ResultsNo Cycling Within-session
CyclingBetween-session Cycling
No Serpentining control condition ? ?
Serpentining ? ? ?
31
ResultsNo Cycling Within-session
CyclingBetween-session Cycling
No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +
accuracy: -familiarity: -usefulness: -change: +freshness: +
Serpentining
32
Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics
Results
33
No Cycling Within-sessionCycling
Between-session Cycling
No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +
accuracy: -familiarity: -usefulness: -change: +freshness: +
#page views: +#interested: +
accuracy: -confusion: +change: +
Serpentining
Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics
34
No Cycling Within-sessionCycling
Between-session Cycling
No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +
accuracy: -familiarity: -usefulness: -change: +freshness: +
#page views: +#interested: +
accuracy: -confusion: +change: +
Serpentining #page views: +#interested: +
accuracy: -familiarity: -usefulness: -
Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics
35
No Cycling Within-sessionCycling
Between-session Cycling
No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +
accuracy: -familiarity: -usefulness: -change: +freshness: +
#page views: +#interested: +
accuracy: -confusion: +change: +
Serpentining #page views: +#interested: +
accuracy: -familiarity: -usefulness: -
too complicated manipulationno interesting sig. results
see the paper for details
Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics
36
No Cycling Within-sessionCycling
Between-session Cycling
No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +
accuracy: -familiarity: -usefulness: -change: +freshness: +
#page views: +#interested: +
accuracy: -confusion: +change: +
Serpentining #page views: +#interested: +
accuracy: -familiarity: -usefulness: -
see the paper
Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics
Some users hate it! L
37
No Cycling Within-sessionCycling
Between-session Cycling
No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +
accuracy: -familiarity: -usefulness: -change: +freshness: +
#page views: +#interested: +
accuracy: -confusion: +change: +
Serpentining #page views: +#interested: +
accuracy: -familiarity: -usefulness: -
see the paper
Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics
Staying users engage more!
38
No Cycling Within-sessionCycling
Between-session Cycling
No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +
accuracy: -familiarity: -usefulness: -change: +freshness: +
#page views: +#interested: +
accuracy: -confusion: +change: +
Serpentining #page views: +#interested: +
accuracy: -familiarity: -usefulness: -
see the paper
Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics
Staying users engage more.
More browsing to get recommendations! L
39
No Cycling Within-sessionCycling
Between-session Cycling
No Serpentining control condition opt out rate: + #page views: + #interested: +interested rate: +
accuracy: -familiarity: -usefulness: -change: +freshness: +
#page views: +#interested: +
accuracy: -confusion: +change: +
Serpentining #page views: +#interested: +
accuracy: -familiarity: -usefulness: -
see the paper
Only significant results are shown.Italic: objective metrics; Non-italic: subjective metrics
Staying users engage more.
Delighted to explore more interesting recommendations! J
Summary
» Within-session cycling• Higher churning risk• Negative subjective perception, but positive
freshness• Higher level of user activities and interested rate
» Between-session cycling or serpentining• Higher level of user activities• Negative effects on subjective perception
40
Efficiency or engagement?
41
Efficiency or engagement?
42
An experience design perspective on recommenders: There is a tradeoff between serving come-and-go users vs. encouraging deeper interaction/engagement !
Messages
» Better understanding of the trade-off between efficiency vs. engagement can help design a better recommender user experience!
» Cycling and serpentining top-N recommendation lists have benefits (higher engagement) but also costs (negative perception)!
» More work combining algorithms and user experience is needed!
43
Thanks! Questions?
» Title: Toward Better Interactions in Recommender Systems: Cycling and Serpentining Approaches for Top-N Item Lists
» Authors: Qian Zhao, Gediminas Adomavicius, F. Maxwell Harper, Martijn Willemsen, Joseph A. Konstan
» Contact• [email protected]• http://www-users.cs.umn.edu/~qian/
44
45