recommender system

Recommender System

hellojinjie2013-06-

19

We will talk about

◦ Netflix Prize

◦ Major challenges

◦ Definitions of subjects and

problems

◦ Recommend methods

◦ Mahout

◦ CNTV 5+ VIP Recommendation

We will not talk about

◦ Architecture of a recommender

system

◦ How to make it robust and

scalability

Netflix Prize

◦ Netflix, Inc. is an American provider of on-demand Internet

streaming media and flat rate DVD-by-mail

◦ 60% of DVDs rented by Netflix are selected based on

personalized recommendations.

Netflix Prize

◦ In October 2006, Netflix released a dataset containing

approximately 100 million anonymous movie ratings and

challenged researchers and practitioners to develop

recommender systems that could beat the accuracy of the

company's recommendation system, Cinematch.

◦ On 21 September 2009, the grand prize of $1,000,000 was

awarded to a team that over performed the Cinematch's

accuracy by 10%.

Major challenges

◦ Data sparsity – 数据庞大；评分分布不均匀。◦ Scalability– 数据庞大；增量更新。◦ Cold start – 新来的用户◦ Diversity vs. accuracy – 不要把路人皆知的推介给我◦ Vulnerability to attacks – 有榜单，就有人刷榜◦ The value of time – 不同时期喜欢不同的东西◦ Evaluation of recommendations – 不同的推介方法谁好谁

差◦ User interface – 优化的展示方式，让用户乐于接受我们的推

介

Evaluation Metrics for Recommendation

◦ The training set ET

-- The training set is treated as known

information

◦ The probe set EP

-- no information from the probe set is

allowed to be used for recommendation.


◦ Accuracy Metrics

◦ Mean Absolute Error (MAE)

◦ Root Mean Squared Error (RMSE)


◦ Precision is the proportion of top

recommendations that are good.

◦ Recall is the proportion of good

recommendations that appear in top

recommendations.

Classifications of recommender systems

◦ Content-based recommendations

◦ Collaborative recommendations

◦ Memory-based collaborative filtering

◦ Standard similarity-based methods

◦ methods employing social filtering

◦ Model-based collaborative filtering

◦ dimensionality reduction methods

◦ diffusion-based methods

◦ Hybrid approaches

Similarity-based methods

◦ User-based recommender

for every other user w

compute a similarity s between u and w

retain the top users, ranked by similarity, as a

neighborhood n

for every item i that some user in n has a preference for,

but that u has no preference for yet

for every other user v in n that has a preference for i

compute a similarity s between u and v

incorporate v's preference for i, weighted by s, into a

running average



DataModel model = new FileDataModel(new File("intro.csv"));

UserSimilarity similarity = new

PearsonCorrelationSimilarity(model);

UserNeighborhood neighborhood =

new NearestNUserNeighborhood(100, similarity,

model);

Recommender recommender =

new GenericUserBasedRecommender(model,

neighborhood, similarity);



• Data model, implemented via DataModel• User-user similarity metric, implemented via UserSimilarity• User neighborhood definition, implemented via

UserNeighborhood• Recommender engine, implemented via a Recommender

(here, GenericUserBasedRecommender)


◦ Item-based recommender

for every item i that u has no preference for yet

for every item j that u has a preference for

compute a similarity s between i and j

add u's preference for j, weighted by s, to a running

average

return the top items, ranked by weighted average


◦ Item-based recommender

DataModel model = new FileDataModel(new File("intro.csv"));

ItemSimilarity similarity = new

PearsonCorrelationSimilarity(model);

Recommender recommender =

new GenericUserBasedRecommender(model,

similarity);

Summary of available recommender implementations

in Mahout

CNTV 5+ VIP Recommendationpassport_260676's preference

( 上半场 11:00) 9- 马竞 - 拉达梅尔 . 法尔考攻入一球 lfp 3.0( 第一节 08:59) 6-EAST- 勒布朗 . 詹姆斯灌篮得分 nba 5.0MV- 即刻出发（演唱：吉克隽逸） nba 3.0

( 第二节 11:00) 24-EAST- 保罗 . 乔治灌篮得分 nba 5.0

userBasedBooleanPref

( 第四节 00:47) 32-WEST- 布雷克 . 格里芬灌篮得分 nba 20.860504 ( 第二节 02:33) 32-WEST- 布雷克 . 格里芬接 24-WEST- 科比 . 布莱恩特传球，灌篮 nba 17.332127

wings nba 9.839406

( 上半场 22:00) 7- 皇家马德里 - 克里斯蒂亚诺 . 罗纳尔多自摆乌龙 lfp 8.962188 托尼 · 帕克现场秀中文 nba 7.3381634 埃文斯再秀创意空中换手 + 飞跃海报 nba 7.2042103 ( 下半场 58:00) 10- 巴塞罗那 - 梅西攻入一球 lfp 7.201148 歌手 NE-YO 劲歌热舞引导东部全明星入场 nba 7.151176 罗斯复制世纪之扣 + 向文斯 · 卡特致敬 nba 7.0464416 ( 第三节 01:25) 34- 掘金 - 贾维尔 . 麦基灌篮得分 nba 6.4302483 http://172.16.0.237:10008/recommend/userID/260676/howMany/10

CNTV 5+ VIP Recommendationpassport_260676's preference

( 上半场 11:00) 9- 马竞 - 拉达梅尔 . 法尔考攻入一球 lfp 3.0( 第一节 08:59) 6-EAST- 勒布朗 . 詹姆斯灌篮得分 nba 5.0MV- 即刻出发（演唱：吉克隽逸） nba 3.0

( 第二节 11:00) 24-EAST- 保罗 . 乔治灌篮得分 nba 5.0

userBasedBooleanPref

( 第四节 00:47) 32-WEST- 布雷克 . 格里芬灌篮得分 nba 20.860504 ( 第二节 02:33) 32-WEST- 布雷克 . 格里芬接 24-WEST- 科比 . 布莱恩特传球，灌篮 nba 17.332127

wings nba 9.839406

( 上半场 22:00) 7- 皇家马德里 - 克里斯蒂亚诺 . 罗纳尔多自摆乌龙 lfp 8.962188 托尼 · 帕克现场秀中文 nba 7.3381634 埃文斯再秀创意空中换手 + 飞跃海报 nba 7.2042103 ( 下半场 58:00) 10- 巴塞罗那 - 梅西攻入一球 lfp 7.201148 歌手 NE-YO 劲歌热舞引导东部全明星入场 nba 7.151176 罗斯复制世纪之扣 + 向文斯 · 卡特致敬 nba 7.0464416 ( 第三节 01:25) 34- 掘金 - 贾维尔 . 麦基灌篮得分 nba 6.4302483

Q&A

References1. Sean Owen, Mahout in Action2. Linyuan Lv, Recommender

Systems

Architecture of NeuRecommendation

IMS etc.

Dispatcher

Recommender Recommender

Data Feeder

Request for recommendation

Dispatch request using round robin

Fetching users’ preferences

Architecture of NeuRecommendation

Data Store Mahout

RPC

Recommender 1. Serve recommendation request

2. Fetch users’ preferences

recommender system

Technology

wings nba

neyo nba

usersimilarity similarity

recommender systemhellojinjie

itemsimilarity similarity

vip recommendation passport

recommendation precision

user w