mlconf - emmys, oscars & machine learning algorithms at netflix
DESCRIPTION
TRANSCRIPT
![Page 1: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/1.jpg)
Emmys, Oscars, and Machine Learning Algorithms @ Netflix Scale November, 2013 Xavier Amatriain Director -‐ Algorithms Engineering -‐ Ne<lix @xamat
![Page 2: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/2.jpg)
“In a simple Netlfix-style item recommender, we would simply apply some form of matrix factorization (i.e NMF)”
![Page 3: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/3.jpg)
What we were interested in: § High quality recommendations
Proxy question: § Accuracy in predicted rating
§ Improve by 10% = $1million!
![Page 4: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/4.jpg)
From the Netflix Prize to today
2006 2013
![Page 5: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/5.jpg)
Big Data @Netflix § > 40M subscribers
§ Ratings: ~5M/day
§ Searches: >3M/day
§ Plays: > 50M/day
§ Streamed hours:
o 5B hours in Q3 2013
Geo-information Time
Impressions Device Info Metadata
Social
Ratings
Demographics
Member Behavior
![Page 6: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/6.jpg)
Smart Models § Regression models (Logistic, Linear, Elastic nets)
§ SVD & other MF models
§ Factorization Machines
§ Restricted Boltzmann Machines
§ Markov Chains & other graph models
§ Clustering (from k-means to HDP)
§ Deep ANN
§ LDA
§ Association Rules
§ GBDT/RF
§ …
![Page 7: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/7.jpg)
Deep-dive into Netflix Algorithms
![Page 8: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/8.jpg)
Rating Prediction
![Page 9: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/9.jpg)
2007 Progress Prize ▪ Top 2 algorithms
▪ SVD - Prize RMSE: 0.8914 ▪ RBM - Prize RMSE: 0.8990
▪ Linear blend Prize RMSE: 0.88
▪ Currently in use as part of Netflix’ rating prediction component
▪ Limitations ▪ Designed for 100M ratings, we have 5B ratings ▪ Not adaptable as users add ratings ▪ Performance issues
![Page 10: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/10.jpg)
SVD - Definition
A[n x m] = U[n x r] Λ [ r x r] (V[m x r])T
▪ A: n x m matrix (e.g., n documents, m terms)
▪ U: n x r matrix (n documents, r concepts)
▪ Λ: r x r diagonal matrix (strength of each ‘concept’) (r: rank of the matrix)
▪ V: m x r matrix (m terms, r concepts)
![Page 11: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/11.jpg)
SVD - Properties ▪ ‘spectral decomposition’ of the matrix:
▪ ‘documents’, ‘terms’ and ‘concepts’:
§ U: document-to-concept similarity matrix
§ V: term-to-concept similarity matrix
§ Λ: its diagonal elements: ‘strength’ of each concept
= x x u1 u2 λ1 λ2
v1 v2
![Page 12: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/12.jpg)
SVD for Rating Prediction § User factor vectors and item-factors vectors
§ Baseline (bias) (user & item deviation from average)
§ Predict rating as
§ SVD++ (Koren et. Al) asymmetric variation w. implicit feedback
§ Where § are three item factor vectors
§ Users are not parametrized
![Page 13: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/13.jpg)
Restricted Boltzmann Machines § Restrict the connectivity in ANN
to make learning easier.
§ Only one layer of hidden units. § Although multiple layers are
possible
§ No connections between hidden units.
§ Hidden units are independent given the visible states..
§ RBMs can be stacked to form Deep Belief Networks (DBN) – 4th generation of ANNs
![Page 14: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/14.jpg)
What about the final prize ensembles? ▪ Our offline studies showed they were too
computationally intensive to scale ▪ Expected improvement not worth the
engineering effort
▪ Plus, focus had already shifted to other issues that had more impact than rating prediction...
![Page 15: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/15.jpg)
Ranking Ranking
![Page 16: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/16.jpg)
Ranking § Ranking = Scoring + Sorting +
Filtering bags of movies for presentation to a user
§ Key algorithm, sorts titles in most contexts
§ Goal: Find the best possible ordering of a set of videos for a user within a specific context in real-time
§ Objective: maximize consumption & “enjoyment”
§ Factors § Accuracy § Novelty § Diversity § Freshness § Scalability § …
![Page 17: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/17.jpg)
Popularity
Pred
icted Ra
Eng
1
2 3
4
5
Linear Model: frank(u,v) = w1 p(v) + w2 r(u,v) + b
Final Ranking Example: Two features, linear model
![Page 18: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/18.jpg)
Popularity
1
2 3
4
5
Final Ranking Pred
icted Ra
Eng
Example: Two features, linear model
![Page 19: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/19.jpg)
Ranking
![Page 20: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/20.jpg)
Learning to Rank ▪ Ranking is a very important problem in many contexts
(search, advertising, recommendations)
▪ Quality of ranking is measured using ranking metrics such as NDCG, MRR, MAP, FPC…
▪ It is hard to optimize machine-learned models directly on these measures ▪ They are not differentiable
▪ We would like to use the same measure for optimizing and for the final evaluation
![Page 21: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/21.jpg)
Learning to Rank Approaches § ML problem: construct ranking model from training data 1. Pointwise (Ordinal regression, Logistic regression, SVM, GBDT, …)
§ Loss function defined on individual relevance judgment
2. Pairwise (RankSVM, RankBoost, RankNet, FRank…) § Loss function defined on pair-wise preferences
§ Goal: minimize number of inversions in ranking
3. Listwise
§ Indirect Loss Function (RankCosine, ListNet…)
§ Directly optimize IR measures (NDCG, MRR, FCP…) § Genetic Programming or Simulated Annealing
§ Use boosting to optimize NDCG (Adarank)
§ Gradient descent on smoothed version (CLiMF, TFMAP, GAPfm @cikm13)
§ Iterative Coordinate Ascent (Direct Rank @kdd13)
![Page 22: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/22.jpg)
Similarity
![Page 23: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/23.jpg)
Similars
![Page 24: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/24.jpg)
What is similarity? § Similarity can refer to different dimensions
§ Similar in metadata/tags § Similar in user play behavior § Similar in user rating behavior § …
§ You can learn a model for each of them and finally learn an ensemble
![Page 25: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/25.jpg)
Graph-based similarities
0.8
0.2
0.3
0.4
0.3
0.7
0.3
![Page 26: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/26.jpg)
Example of graph-based similarity: SimRank
▪ SimRank (Jeh & Widom, 02): “two objects are similar if they are referenced by similar objects.”
![Page 27: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/27.jpg)
Final similarity ensemble § Come up with a score of play similarity, rating similarity,
tag-based similarity…
§ Combine them using an ensemble § Weights are learned using regression over existing response
§ The final concept of “similarity” responds to what users vote as similar
![Page 28: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/28.jpg)
Row
Sel
ectio
n
![Page 29: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/29.jpg)
Row SelecEon Inputs § Visitor data
§ Video history
§ Queue adds
§ RaEngs
§ Taste preferences
§ Predicted raEngs
§ PVR ranking
§ etc.
§ Global data § Metadata § Popularity § etc.
![Page 30: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/30.jpg)
Row SelecEon Core Algorithm
Candidate data Groupify Score/Select Group
selecEon
SelecEon constraints
ParEal selecEon
![Page 31: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/31.jpg)
SelecEon Constraints • Many business rules and heurisEcs
– RelaEve posiEon of groups – Presence/absence of groups – Deduping rules – Number of groups of a given type
• Determined by past A/B tests • Evolved over Eme from a simple template
![Page 32: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/32.jpg)
GroupificaEon • Ranking of Etles according to ranking algo
• Filtering • Deduping • Forma^ng
• SelecEon of evidence • ...
Candidate data
First n selected groups
Group for posiEon n
+1
![Page 33: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/33.jpg)
Scoring and SelecEon • Features considered – Quality of items
– Diversity of tags – Quality of evidence – Freshness – Recency – ...
– ARO scorer – Greedy algorithm
– Framework supports other methods § Backtracking § Approximate integer
programming
§ etc.
![Page 34: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/34.jpg)
Search Recommendations
![Page 35: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/35.jpg)
Search Recommendation CombinaEon of Different models
• Play Aaer Search: TransiEon on Play aaer Query • Play Related to Search: Similarity between user’s (query,
play) • …
Read this paper for a description of a similar system at LinkedIn
![Page 36: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/36.jpg)
Unavailable Title Recommendations
![Page 37: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/37.jpg)
Gamification Algorithms
![Page 38: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/38.jpg)
rating game Mood Selection Algorithm 1. Editorially selected moods 2. Sort them according to users consumption 3. Take into account freshness, diversity...
Rating Game
![Page 39: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/39.jpg)
More data or better models?
![Page 40: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/40.jpg)
More data or better models?
Really?
Anand Rajaraman: Former Stanford Prof. & Senior VP at Walmart
![Page 41: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/41.jpg)
Sometimes, it’s not about more data
More data or better models?
![Page 42: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/42.jpg)
[Banko and Brill, 2001]
Norvig: “Google does not have better Algorithms, only more Data”
Many features/ low-bias models
More data or better models?
![Page 43: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/43.jpg)
More data or better models?
Sometimes, it’s not about more data
![Page 44: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/44.jpg)
“Data without a sound approach = noise”
![Page 45: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/45.jpg)
Conclusion
![Page 46: MLConf - Emmys, Oscars & Machine Learning Algorithms at Netflix](https://reader033.vdocuments.net/reader033/viewer/2022051819/54c65a224a795971538b4579/html5/thumbnails/46.jpg)
More data + Smarter models +
More accurate metrics + Better system architectures
Lots of room for improvement!