user-centric evaluation of a k-furthest neighbor collaborative filtering recommender algorithm
DESCRIPTION
Collaborative filtering recommender systems often use nearest neighbor methods to identify candidate items. In this paper we present an inverted neighborhood model, k-Furthest Neighbors, to identify less ordinary neighborhoods for the purpose of creating more diverse recommendations. The approach is evaluated two-fold, once in a traditional information retrieval evaluation setting where the model is trained and validated on a split train/test set, and once through an online user study (N=132) to identify users’ erceived quality of the recommender. A standard k-nearest neighbor recommender is used as a baseline in both evaluation settings. our evaluation shows that even though the proposed furthest neighbor model is outperformed in the traditional evaluation setting, the perceived usefulness of the algorithm shows no significant difference in the results of the user study.TRANSCRIPT
Technische Universität Berlin
1 @alansaid 2 @alsothings
User-Centric Evaluation of a K-Furthest Neighbor Collaborative Filtering Recommender Algorithm Alan Said*1, Ben Fields#2, Brijnesh J. Jain*, Sahin Albayrak*
February 27th, 2013
*
# musicmetric
CSCW 2013, San Antonio, TX, USA
Abstract
• New recommendation algorithm for diverse recommendations
• Based on the k-nearest neighbor algorithm
• Two types of evaluation
o standard offline evaluation
o user-centric online evaluation
• Proposed algorithm performs worse than baseline in offline evaluation but has higher perceived usefulness from the users in online evaluation
February 27th, 2013 2
Outline
February 27th, 2013
• Background
• Recommendation
• K-Nearest Neighbors (knn)
• K-Furthest Neighbors (kfn)
• Evaluation & Results
• Conclusions
3
Background and Acknowledgements
Started as a (not very serious) discussion at IJCAI & ICWSM 2011
• Ben Fields - @alsothings
• Òscar Celma - @ocelma
• Markus Schedl - @m_schedl
• Mohamed Sordo - @neomoha
February 27th, 2013 4
Recommendation
• What is it?
o Personalized information filtering
• What is the difference to search?
o Implicit
o Passively finds most interesting items
• How?
February 27th, 2013 5
Recommendation - An example (knn)
Recommending a movie to Bert:
what/who Bert Ernie Big Bird Cookie
Monster Elmo
Herry
Monster
Toy Story 4 4 5 1 4
E.T. 2 5 2
Beetlejuice 4 4 5 2 3
Shrek 1 3 1
Zoolander 4 1
February 27th, 2013 6
Recommendation - An example (knn)
Recommending a movie to Bert:
what/who Bert Ernie Big Bird Cookie
Monster Elmo
Herry
Monster
Toy Story 4 4 5 1 4
E.T. 2 5 2
Beetlejuice 4 4 5 2 3
Shrek 1 3 1
Zoolander 4 1
Similar to Bert
Potential movies
to recommend
poor
rating
recommendation
K-Nearest Neighbor
February 27th, 2013 6
Recommendation - A counter example
What happens if we flip it?
Can we recommend movies
disliked by those who are
dissimilar to Bert?
Yes!
February 27th, 2013 7
Recommendation - A counter example (kfn)
Recommending a movie to Bert:
what/who Bert Ernie Big Bird Cookie
Monster Elmo
Herry
Monster
Toy Story 4 4 5 1 4
E.T. 2 5 2
Beetlejuice 4 4 5 2 2 3
Shrek 1 3 1
Zoolander 4
1. Who is dissimilar to Bert?
February 27th, 2013 8
what/who Bert Cookie
Monster
Toy Story 4 1
E.T.
Beetlejuice 4 2
Shrek 1
Zoolander
Recommendation - A counter example (kfn)
Recommending a movie to Bert:
1. Who is dissimilar to Bert?
Disliked by Cookie Monster -
Liked by Bert?
2. What do they dislike?
K-Furthest Neighbor
February 27th, 2013 9
Evaluation
What are the effects of this?
Diversity : • Less popular items
• Items the users are not familiar with
• Non standard items
February 27th, 2013 10
Evaluation - Recommendation Accuracy
Traditional - Offline Evaluation
• Movielens 10M, 70k users
• Precision@N for users with >2N ratings
• Furthest performs at ~60% of Nearest neighbor (for N=100)
However
• lists of recommended items are practically
disjoint
February 27th, 2013 11
<0.001
Evaluation
12 February 27th, 2013
•Are we missing something?
Yes
train
test
Evaluation - Online User Study
February 27th, 2013 13
Evaluation - Online User Study
10 recommended
movies
7 questions
February 27th, 2013 13
Evaluation – Recommendation Utility
Data
• 132 users
• 10 recommended movies each
• knn: 47 users
• kfn: 43 users
• random: 42 users
• training set: Movielens 10M
Questions • Novel? • Obvious? • Recognizable? • Serendipitous? • Useful? • Best movie? • Worst movie? • Rate each seen movie • State whether movie is familiar • State whether you would see it
February 27th, 2013 14
Evaluation – Recommendation Utility
Do you know the movie?
February 27th, 2013 15
Evaluation – Recommendation Utility
Have you seen it?
February 27th, 2013 15
Evaluation – Recommendation Utility
Would you watch it?
February 27th, 2013 15
Evaluation – Recommendation Utility
rating novelty obviousness recognizable serendipity usefulness
knn 3.64 3.83 2.27 2.69 2.71 2.69
kfn 3.65 3.95 1.79 2.07 2.65 2.63
random 3.07 4.17 1.64 1.81 2.48 2.24
highest rating less obvious/recognizable comparable serendipity and
usefulness
remember: knn and kfn recommend different items, still the experienced quality is similar (or higher)
February 27th, 2013 16
Likert scale 1: least agree; 5: most agree
Conclusion
Recommending what your anti-peers do not like creates:
• more diverse recommendations,
• with comparable overall usefulness,
• even though standard offline evaluation says otherwise
February 27th, 2013 17
Questions?
Thank you for listening!
For more RecSys stuff, check out:
www.recsyswiki.com
February 27th, 2013 18