recommender system introduction
Post on 15-Jan-2015
Embed Size (px)
- 1. Recommender SystemIntroductionxiangliang@hulu.com
2. What is good recommender system? 3. Outline What is recommender system? Mission History Problems What is good recommender system? Experiment Methods Evaluation Metric 4. Information Overload 5. How to solve information overload Catalog Yahoo, DMOZ Search Engine Google, Bing 6. Mission Help user find item of their interest. Help item provider deliver their item toright user. Help website improve user engagement. 7. Recommender System 8. Search Engine vs. Recommender System User will try search engine if they have specific needs they can use keywords to describe needs User will try recommender system if they do not know what they want now they can not use keywords to describe needs 9. History: Before 1992 Content Filtering An architecture for large scale informationsystems  (Gifford, D.K) MAFIA: An active mail-filter agent for anintelligent document processing support (Lutz, E.) A rule-based message filtering system (Pollock, S. ) 10. History: 1992-1998 Tapestry by Xerox Palo Alto  First system designed by collaborative filtering Grouplens  First recommender system using rating data Movielens  First movie recommender system Provide well-known dataset for researchers 11. History: 1992-1998 Fab : content-based collaborativerecommendation First unified recommender system Empirical Analysis of Predictive Algorithmsfor Collaborative Filtering  (John S.Breese) Systematically evaluate user-basedcollaborative filtering 12. History: 1999-2005 Amazon proposed item-based collaborativefiltering (Patent is filed in 1998 and issuedin 2001) [link] Thomas Hofmann proposed pLSA and apply similar method on collaborativefiltering  Pandora began music genome project 13. History: 1999-2005 Lastfm using Audioscrobbler to generateuser taste profile on musics. Evaluating collaborative filteringrecommender systems  (Jonathan L.Herlocker) 14. History: 2005-2009 Toward the Next Generation ofRecommender Systems: A Survey of theState-of-the-Art and Possible Extensions. (Alexander Tuzhilin) Netflix Prize [link] Latent Factor Model (SVD, RSVD, NSVD, SVD++) Temporal Dynamic Collaborative Filtering Yehuda Koren [link]s team get prize 15. History: 2005-2009 ACM Conference on Recommender System (Minneapolis, Minnesota, USA) Digg, Youtube try recommender system. 16. History: 2010-now Context-Aware Recommender Systems Music Recommendation and Discovery Recommender Systems and the Social Web Information Heterogeneity and Fusion inRecommender Systems Human Decision Making in Recommender Systems Personalization in Mobile Applications Novelty and Diversity in Recommender Systems User-Centric Evaluation 17. History: 2010-now Facebook launches instant personalization Clicker Bing Trip Advisor Rotten Tomatoes Pandora 18. Problems Main Problems Top-N Recommendation Rating Prediction 19. Problems Top-N Recommendation InputuseritemAaBaBb Output 20. Problems Top-N Recommendation Inputuseritem ratingAaBaBb Output 21. What is good recommender system? 22. Experiment Methods Offline Experiment User Survey Online Experiment AB Testing 23. Experiment Methods Offline ExperimentDataSetTrainTest Advantage: Only rely on dataset Disadvantage: Offline metric can not reflect business goal 24. Experiment Methods User Survey Advantage: Can get subjective metrics Lower risk than online testing Disadvantage: Higher cost than offline experiments Some results may not have statistical significance Users may have different behaviors under testingenvironment or real environment Its difficult to design double blink experiments. 25. Experiment Methods On line experiments (AB Testing) Advantage: Can get metrics related to business goal Disadvantage: High risk/cost Need large user set to get statistical significant result 26. Experiment Metrics User Satisfaction Prediction Accuracy Coverage Diversity Novelty Serendipity Trust Robust Real-time 27. Experiment Metrics User Satisfaction Subjective metric Measured by user survey or online experiments 28. Experiment Metrics Prediction Accuracy Measured by offline experiments Top-N Recommendation Precision / Recall Rating Prediction MAE, RMSE 29. Experiment Metrics Coverage Measure the ability of recommender system torecommend long-tail items.| R (u , N ) | u UCoverage |I| Entropy, Gini Index 30. Experiment Metrics Diversity Measure the ability of recommender system tocover users different interests. Different similarity metric generate differentdiversity metric. 31. Experiment Metrics Diversity (Example) Watch History Related Items 32. Experiment Metrics Novelty Measure the ability of recommender system tointroduce long tail items to users. International Workshop on Novelty andDiversity in Recommender Systems [link] Music Recommendation and Discovery in theLong Tail [Oscar Celma] 33. Experiment Metrics Serendipity A recommendation result is serendipity if: its not related with users historical interest its novelty to user user will find its interesting after user view it 34. Experiment Metrics Trust If user trust recommender system, they willinteract with it. Ways to improve trust: Transparency Social Trust System (Epinion) 35. Experiment Metrics Robust The ability of recommender system to preventattack. Neil Hurley. Tutorial on Robustness ofRecommender System. ACM RecSys 2011. 36. Experiment Metrics Real-time Generate new recommendations when userhave new behaviors immediately. 37. Too many metric!Which is most important? 38. How to do trade-off Business goal Our belief Making new algorithms by 3 stepsexperiments: Offline testing User survey Online testing 39. Thanks!