Goal-driven Collaborative FilteringA Directional Error Based Approach

Tamas Jambor and Jun Wang

University College London

Structure of the talk

• Background/Problem description• Goal-driven design• Experimental results• Conclusions

Collaborative filtering

• Predicting user preference

towards unknown items• Based on previously expressed preferences

Love Actually

Pulp Fiction

Crazy Heart

WhiteRibbon

Up in the Air

A SingleMan

Sophie «««« « ««««¶ «««««

Peter ««««« «««¶¶ ««¶¶¶

Jaden «««¶¶ «««¶¶ «««««

?? ?

? ? ?

? ? ?

Evaluation metrics

• Root Mean Squared Error• Netflix recommendation competition

adopted this metric• The objective function for some of the SVD

implementations is equivalent to the performance measure [Koren et al 2009]

• Criticism– Error criterion is uniform across rating scales– Is it consistent with users’ satisfactions?

))ˆ(( 2rrE

Goal-driven design

• We argue that – Measure does not always reflect user needs– Different user needs require different performance

measures

• The algorithm should be defined based on user needs– Start from the user point of view, define measure and

algorithm accordingly

Rating-prediction error offset (SVD)

Observed 1Predicted 3

Observed 5Predicted 3

Observed 3Predicted 1

Observed 3Predicted 5

5««««««¶¶¶¶ «««¶¶

31

Boundaries and the direction of error

• Taste boundary - interval between liked and disliked items

• Direction – error towards the boundary• Magnitude – whether the error crosses taste

boundary

Directional risk preference of prediction

The two dimensional weighting function

r = 1,2 r = 3 r = 4,5

p <= 2.5 w1 w2 w3

2.5<p<=3.5 w4 w5 w6

P > 3.5 w7 w8 w9

Two-stage Optimization (in General)

Learning the Directional

Errors

Learning the Recom. Model

Testing

Feedback/IR Metrics

Two-stage Optimization (An example)

Directional Errors

Modeling

Rec. Model (SVD)Evaluations

NDCG/MRR

argminq, p

w(rui qiT

ui pu )

2 ( qi2 pu

2)

Genetic algorithmNDCG as fitness function

Plug in the learned Weights in SVD Training

Genetic algorithms

• Search algorithms that work via the

process of natural selection• Start with a sample set of potential solutions (a set

of weights)• Evolve towards a set of more optimal solutions• Poor solutions tend to die out (smaller NDCG)• Better solutions remain in the population (higher

NDCG)

Experiments

• MovieLens 100k dataset• 1862 movies, 943 users• Only using ratings• Five-fold cross validation

Evaluation metrics

• Recommendation as a ranking problem• IR measures

– Normalized discounted cumulative gain (NDCG)– Mean average precision (MAP)– Mean reciprocal rank (MRR)

Results – Experiment I

SVD with weights where w7>w8>w4

r = 1,2 r = 3 r = 4,5

p <= 2.5 0.0759 0.0407 0.0264

2.5<p<=3.5 0.0837 0.1676 0.2381

p > 3.5 0.0125 0.0583 0.2966

Baseline SVD

r = 1,2 r = 3 r = 4,5

p <= 2.5 0.0517 0.0193 0.0106

2.5<p<=3.5 0.0904 0.1461 0.1391

p > 3.5 0.0299 0.1012 0.4115

Results – Experiment II

r = 1,2 r = 3 r = 4,5

p <= 2.5 w1 w2 w3

2.5<p<=3.5 w4 w5 w6

P > 3.5 w7 w8 w9

Results – Experiment II

• Genetic algorithm to find optimal weigh for sector w7,w8 and w4 (statistically significant)

Weighted Baseline

MAP 0.450 0.447

MRR 0.899 0.889

NDCG@10 0.726 0.720

NDCG@5 0.574 0.570

NDCG@3 0.450 0.447

Probability of correct prediction within sectors

Probability of predicting non-relevant items relevant

Improved user experience

• More likely to receive relevant items on their recommendation list

• Less likely that lower rated items receive higher predictions

• But it is more likely that higher rated items receive lower predictions

Conclusion

• Optimize algorithm from the user point of view• Identify directional errors• Assign risk to each direction• Approach can be changed depending on how

items are presented

Future work

• Taste boundaries might be user dependent • Directional error across items or users• Different recommender goals

Thank you.

References

• Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1) (2004)

• Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic framework for performing collaborative filtering. In: SIGIR '99. (1999)

• Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8) (2009)

• Wang, J., de Vries, A.P., Reinders, M.J.T.: Unifying user-based and item-based collaborative filtering approaches by similarity fusion. In: SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, New York, NY, ACM Press