RANKING AND DIVERSITY IN RECOMMENDATIONS Alexandros Karatzoglou @alexk_z

Thanks

Linas Baltrunas Telefonica

Saul Vargas Universidad Autonoma de Madrid

Yue Shi TU Delft

Pablo Castells Universidad Autonoma de Madrid

Telefonica Research in Barcelona • User Modeling: Recommender Systems • Data Mining, Machine Learning • Multimedia Indexing and Analysis • HCI • Mobile Computing • Systems and Networking

•  http://www.tid.es

Recommendations in Telefonica

People You May Know@Tuenti

Recommendations in Telefonica

Firefox OS Marketplace

Collaborative Filtering

+

+

+ +

+

+

+

+

+

fij = hUi,Mjifij = hUiMji+X

k2Fi

↵ik

|Fi|hUkMji

Tensor Factorization HOSVD CP-Decomposition

Fijk = S ⇥U Ui ⇥M Mj ⇥C CkFijk = hUi,Mj , CkiKaratzoglou et al. @ Recsys 2010

Publications in Ranking CIKM 2013: GAPfm: Optimal Top-N Recommendations for Graded Relevance Domains RecSys 2013: xCLiMF: Optimizing Expected Reciprocal Rank for Data with

Multiple Levels of Relevance RecSys 2012: CLiMF: Learning to Maximize Reciprocal Rank with Collaborative

Less-is-More Filtering * Best Paper Award SIGIR 2012: TFMAP: Optimizing MAP for Top-N Context-aware Recommendation Machine Learning Journal, 2008: Improving Maximum Margin Matrix Factorization

* Best Machine Learning Paper Award at ECML PKDD 2008 RecSys 2010: List-wise Learning to Rank with Matrix Factorization for Collaborative Filtering NIPS 2007: CoFiRank - Maximum Margin Matrix Factorization for Collaborative Ranking

Recommendations are ranked lists

Popular Ranking Methods •  In order to generate the ranked item list, we need some

relative utility score for each item •  Popularity is the obvious baseline •  Score could depend on the user (personalized) •  Score could also depend on the other items in the list (list-wise)

• One popular way to rank the items in RS is to sort the items according to the rating prediction •  Works for the domains with ratings •  Wastes the modeling power for the irrelevant items

Graphical Notation

Relevant  

Irrelevant  

Irrelevant  

Irrelevant  

Irrelevant  

Irrelevant  

Relevant  

Relevant  

Ranking using latent representation

•  If user = [-100, -100] •  2d latent factor •  We get the corresponding ranking

Matrix Factorization (for ranking) • Randomly initialize item vectors • Randomly initialize user vectors • While not converged

•  Compute rating prediction error •  Update user factors •  Update item factors

•  Lets say user is [-100, -100] •  Compute the square error

•  (5-<[-100, -100], [0.180, 0.19]>)2=1764 •  Update the user and item to the direction

where the error is reduced (according to the gradient of the loss)

8 items with ratings and random factors

Learning: Stochastic Gradient Descent with Square Error Loss

Square Loss User: [3, 1], RMSE=6.7

Learning  to  Rank  for  Top-­‐k  RecSys  • Usually  we  care  about  accurate  ranking  and  not  ra=ng  predic=on  •  Square  Error  loss  op=mizes  to  accurately  predict  1s  and  5s.  

•  RS  should  get  the  top  items  right  -­‐>  Ranking  problem  • Why  not  to  learn  how  to  rank  directly?  

•  Learning  to  Rank  methods  provide  up  to  30%  performance  improvements  in  off-­‐line  evalua=ons  

•  It  is  possible,  but  a  more  complex  task  

Example: average precision (AP)

AP =

|S|X

k=1

P (k)

|S|

• AP: we compute the precision at each relevant position and average them

P@1+P@2+P@43

=1/1+ 2 / 2+3 / 4

3= 0.92

Why is hard? Non Smoothness Example: AP

u:[-20,-20]

u:[20,20]

AP vs RMSE

The Non-smoothness of Average Precision

APm =1

PNi=1 ymi

NX

i=1

ymi

rmi

NX

j=1

ymjI(rmj rmi)

AP =

|S|X

k=1

P (k)

|S|

ymi

rmi

I(·)is 1 if item i is relevant for user m and 0 otherwise

indicator function (1 if it is true, 0 otherwise)

Rank of item i for user m

How can we get a smooth-AP? • We  replace  non  smooth  parts  of  MAP  with  smooth  approxima=on  

g(x) = 1/(1 + e

�x)

1

rmi⇡ g(fmi) = g(hUm, Vii)

How can we get a smooth-MAP? • We  replace  non  smooth  parts  of  MAP  with  smooth  approxima=on  

g(x) = 1/(1 + e

�x)

I(rmj rmi) ⇡ g(fmj � fmi) = g(hUm, Vj � Vii)

u:[-20,-20]

u:[20,20]

Smooth version of MAP

Sometimes approximation is not very good…

Ranking Inconsistencies • Achieving a perfect ranking for all users is not possible

•  Two Sources of Inconsistencies:

•  1) Factor Models (all models) have limited expressive power and cannot learn the perfect ranking for all users

•  2) Ranking functions approximations are inconsistent e.g. A >B & B>C but C > A

Summary on Ranking 101

Area Under the ROC Curve (AUC)

AUC :=1

|S+||S�|

S+X

i

S�X

j

I(Ri < Rj)

Reciprocal Rank (RR)

RR :=1

Ri

Average Precision (AP)

AP =

|S|X

k=1

P (k)

|S|

AP vs RR

DCG =X

i

2score(i) � 1

log2(i+ 2)

Normalized Discounted Cumulative Gain (nDCG)

Relevance solved! Is that all?

•  Ranking “solves” the relevance problem

•  Can we be happy with the results?

Relevance solved! Is that all?

•  Coverage

•  Diversity

•  Novelty

•  Serendipity

Diversity in Recommendations

•  Diversity using Genres

•  Movies, Music, Books

•  Diversity should fulfill:

•  Coverage •  Redundancy •  Size Awareness

Diversity methods for RecSys

•  Topic List Diversification or •  Maximal Margin Relevance

fMMR(i;S) = (1� �) rel(i) + � minj2S

dist(i, j)

Diversity methods for RecSys

•  Intent-Aware IR metrics

ERR� IA =X

s

p(s)ERR

Example

Action Comedy Sci-Fi Western

Action Thriller Sci-Fi Western

Adventure Western

Genres 154

639

870

46

187146

1258

60

30

232

75202

171

8

21

37116

9

9

11

10

86

43

42

4

1

0

Action (517)

Comedy (1267)

Drama (1711)

Romance (583)

Thriller (600)

Genres and Popularity

Binomial Diversity

•  We base a new Diversity Metric on the Binomial Distribution

P (X = k) =

✓N

k

◆pk(1� p)N�k

User Genre Relevance

•  Fraction of Items of genre “g” user interacted with

•  Global fraction of items of genre “g”

•  Mix

p00g =kIug

|Iu|

p0g =

Pu k

IugP

u |Iu|

pg = (1� ↵) p0g + ↵ p00g

Coverage

Coverage(R) =Y

g/2G(R)

P (Xg = 0)1/|G|

•  Product of the probabilities of the genres not represented in the list not being picked by random

Non-Redundancy

P (Xg � k | Xg > 0) = 1�k�1X

l=1

P (Xg = l | Xg > 0)

NonRed(R) =Y

g2G(R)

P (Xg � k

Rg | Xg > 0)1/|G(R)|

Non-Redundancy

Binomial Diversity

BinomDiv(R) = Coverage(R) ·NonRed(R)

Re-Ranking •  Re-Rank based on Relevance and Binomial Diversity

f

BinomDiv

(i;S) = (1� �) normrel

(rel(i)) + � norm

div

(div(i;S))

normX(x) =x� µX

�X

Example

Thanks! • Questions?