1

Recsys12 && KDD 12 Brief Summary

Xiwang Yang

1

the 6th ACM International Conference on Recommender Systems (RecSys 2012)

RecSys 2012 Conference Introduction Evaluation Metrics Learning to Rank Social RS Context-aware and location-based

recommendations HCI, User-centric, interfaces & explanations System Design

2

RecSys12 Introduction

Premier global forum for discussing the state of the art in recommender System

Long paper acceptance rate: 24/119 = 20% Single Track > 270 attendence 1/3 from industry

3

Evaluation Metrics Workshop:Recommendation Utility Evaluation: Beyond RMSE

Organizer: Xavier Amatriain (Netflix), Harald Steck (Netflix), Pablo Castells (UAM), Arjen de Vries, and Christian Posse (LikedIn)

Specific questions that the workshop aims to address include the following:

What are the unmet needs and challenges for evaluation in the RS field? What changes would we like to see? How could we speed up progress?

What relevant recommendation utility and quality dimensions should be cared for? How can they be captured and measured?

How can metrics be more clearly and/or formally related to the task, contexts and goals for which a recommender application is deployed?

How should IR metrics be applied to recommendation tasks? What aspects require adjustment or further clarification? What further methodologies should we draw from other disciplines (HCI, Machine Learning, etc.)?

4

Evaluation Metrics Workshop:Recommendation Utility Evaluation: Beyond RMSE

Can we predict the success of a recommendation algorithm with our offline experiments? What offline metrics correlate better and under which conditions?

What are the outreach and limitations of offline evaluation? How can online and offline experiments complement each other?

What type of public datasets and benchmarks would we want to have available, and how can they be built?

How can the recommendation effect be traced on business outcomes?

How should the academic evaluation methodologies improve their relevance and usefulness for industrial settings?

How do we envision the evaluation of recommender systems in the future?

5

Evaluation Metrics Industry Keynote: Ron Kohavi (Microsoft): Online Controlled

Experiments: Introduction, Learnings, and Humbling Statistics Ron Kohavi, General Manager Experimentation Platform,

Microsoft Controlled experiments at Microsoft Bing, very good work, 

2012 kdd paper;

6

Controlled Experiments in One Slide

Concept is trivial Randomly split traffic between

two (or more) versions• A (Control)• B (Treatment)

Collect metrics of interest Analyze  

7

100%Users

50%Users

50%Users

Control:Existing System

Treatment:Existing System with Feature X

Users interactions instrumented, analyzed & compared

Analyze at the end of the experiment

Must run statistical tests to confirm differences are not due to chanceBest scientific way to prove causality, i.e., the changes in metrics are caused by changes introduced in the treatment(s)

Evaluation Metrics

Session: Multi-Objective Recommendation and Human Factors Multiple Objective Optimization in Recommendation

Systems Mario Rodriguez and others explain how they design

LinkedIn recommendations by optimizing to several objectives at once (e.g. candidate that is good for the job + who is open to new opportunities). They report results from an AB Test run on LinkedIn

8

Evaluation Metrics Session: Multi-Objective Recommendation and Hu

man Factors Pareto-Efficient Hybridization for Multi-Objective Re

commender Systems• Marco Tulio Ribeiro-Universidade Federal de Minas Gerais &

Zunnit Technologies• The problem of combining recommendation algorithms

grows significantly harder when multiple objectives are considered simultaneously.

• take the multi-objective a step further. In their case, they optimize the system to not only be accurate, but also present novel or diverse items.

9

Outline

Learning to Rank• Session: Top-N Recommendation• Social top-k RS:• Industry invited talk: Ralf Herbrich (Facebook): Distrib

uted, Real-Time Bayesian Learning in Online Services

10

Learning to Rank Focus more on ranking than rating prediction NOW! Session: Top-N Recommendation

CLiMF: Learning to Maximize Reciprocal Rank with Collaborative Less-is-More Filtering

• Best paper, Yue Shi, Delft, intern at Telefonica• Optimize Mean Reciprocal Rank (MRR) directly. • MRR is a well-known information retrieval metric for

measuring the performance of top-k recommendations Similar work: "

TFMAP: Optimizing MAP for top-n context-aware recommendation

• SIGIR 2012. • Optimize Top-N, Mean Average Precision • Uses tensor factorization to model implicit feedback data

(e.g.,purchases, clicks) with contextual information• fast learning algorithm

11

Learning to Rank Focus more on ranking than rating prediction NO

W! Session: Top-N Recommendation

"Ranking with Non-Random Missing Ratings: Influence of Popularity and Positivity on Evaluation Metrics

• An interesting study on the very important issue of negative sampling, and popularity bias in learning to rank. The paper discusses these effects on the AUC (Area Under the Curve) measure.

12

Learning to Rank Session: Top-N Recommendation

"Sparse Linear Methods with Side Information for Top-N Recommendations“

• University of Minnesota in the Twin Cities• multidimensional context-aware learning to rank

Alternating Least Squares for Personalized Ranking• Gravity R&D• Dense math• invited anyone not interested in Mathematics to leave the room

• proposed a computationally efficient ranking based method RankALS that optimizes the original objective function, without sampling.

"On Top-k Recommendation Using Social Networks”13

Learning to Rank "Real-Time Top-N Recommendation in Social Streams

University of Hannover Focus on analyzing social streams(twitter) in real-time

for personalized topic recommendation and discovery. Industry invited talk: Ralf Herbrich (Facebook): Distributed,

Real-Time Bayesian Learning in Online Services Bayesian Factor Models for large-scale distributed ranking The same author and others from MSR named it as "

Matchbox“, is now used in different settings Poster "The Xbox Recommendation System“

Apply matchbox to recommending movies and games for the Xbox Poster “Collaborative Learning of Preference Rankings”

Erasmus School of Economics & MSR apply it to sushi recommendation

14

Social RS Session: Social Recommendation

"Spotting Trends: The Wisdom of the Few” Wisdom of the Few, using a reduced set of experts

for recommendations

Popular != Trending

iCoolhunt users are encouraged to take pictures of objects that they think ‘cool’, upload them and share them with friends online.

15

Social RS Workshop on Recommender Systems and the Social W

eb Extending FolkRank with Content Data Leveraging Publication Metadata and Social Data into

FolkRank for Scientific Publication Recommendation Context Determines Content - An Approach to Resourc

e Recommendation in Folksonomies FReSET - An Evaluation Framework for Folksonomy-ba

sed Recommender Systems Aggregating Content and Network Information to Cura

te Twitter User Lists Online Dating Recommender Systems: The Split-Comp

lex Number Approach Social Media-Driven News Personalization  Trust-Based Local and Social Recommendation

16

Context-aware and location-based recommendations

Workshop: Personalizing the Local Mobile Experience

Workshop on Context-Aware Recommender Systems

Session: Contextual and Semantically Aware Recommendation Context-Aware Music Recommendation Based on

Latent Topic Sequential Patterns",• playlist generation

"Ads and the City: Considering Geographic Distance Goes a Long Way”

• location-aware recommendations.

17

HCI, User-centric, interfaces & explanations Tutorial: Conducting User Experiments in

Recommender Systems • Bart Knijnenburg, UCI• overview of how to conduct user studies for

recommender systems Paper: TasteWeights: A Visual Interactive Hybrid

Recommender System• USSB

Paper: Inspectability and Control in Social Recommenders

• Bart Knijnenburg, UCI• Analyze the effect of giving more information and

control to users in the context of social recommendations.

Workshop on Interfaces for Recommender System.

18

System Design

Tutorial: Building Industrial-scale Real-world Recommender Systems

• Xavier Amatriain, Netflix Mendeley Suggest: Engineering a Personalized

Article Recommendation System• Kris Jack from Mendeley• he explained how they make use of AWS and Mahout

in a system that can generate personalized recommendations for about $60 a month

From a toolkit of recommendation algorithms into a real business

• Domonkos Tikk from Gravity R&D• evolved from being a team in the Netflix Prize to a

real-world company with very interesting projects

19

RecSys in China- Booming RecSys China

Technical Community, ~5000 members RecSys 13 in HongKong Baidu

• RecSys team built two years ago, size: ~100 Taobao

• RecSys team: > 100 Weibo

• RecSys team: ~20 Tencent Huawei

• Noah's Ark Research Lab in Hong Kong• Built July, 2012• long term ~100 researchers

20

How user evaluate each other in social media

Keynote-Jure Leskovec-Stanford University

21

22

23

KDD2012

August12-16, Beijing, China First held in Asian Attendance: 1232 Long Paper Acceptance Rate: 133/755 = 17.6% Three Research Tracks about Recommendation Research Session : Personalization and

Recommendation Research Session: Ads and Video Recommendation Research Session: Recommendation Research Session : Matrices and Tensors

24

Mining Heterogeneous Information Networks- Jiawei Han - UIUC

25

26

27

28

29

30

31

32

33

Big Data Panel Discussion debate on the following questions:

What is the nature of Big Data? What are the Big Data problems that you have encountered? Is this a long-term challenge or a short-term fad?

What opportunities and challenges does data mining face on Big Data?

What are effective Big Data solutions? What platforms, sampling solutions, and applications are most effective for handling Big Data?

34

Some of the Opinions Christos Faloutsos – CMU

Large data size: > 100 machines, > 1 Tera Bytes Jiawei Han

Large Complexity Michael I. Jordan

? Sampling everything

Some other voice Big data rate in HFT

35