data over matter: innovating the next generation of products
TRANSCRIPT
Data Over Matter
| Eli Bressert Manager, Data Engineering & Analytics
Innovating the next generation of products
today's talkEvolution of Netflix
Data at Netflix
Data over matter
evolution of Netflix
1999 DVD Service 2007 Streaming Service 2012 Netflix Originals 2016 Netflix Global
1999 DVD Service 2007 Streaming Service 2012 Netflix Originals 2016 Netflix Global
1999 DVD Service 2007 Streaming Service 2012 Netflix Originals 2016 Netflix Global
2012 2013 2014 2015 2016
1999 DVD Service 2007 Streaming Service 2012 Netflix Originals 2016 Netflix Global
1999 DVD Service 2007 Streaming Service 2012 Netflix Originals 2016 Netflix Global
data @ Netflix
data over
matter
experimentation not opinion
experimentation not opinion
user experience in the UI
2011
2013
image optimization
a/b testing @ Netflix
Discovery Algorithms Design/UI Product Functionality Messaging Streaming qualityM
NM Sign-up flow Messaging Marketing
OtherNetflixServices
architecture overview
Ignite org structure
A/B test data from multiple teams common UI layer & shared resources A/B test visualization is decoupled from A/B test data processing layer
additional supportBYOD (Bring Your Own Data) Internal Engineers 3rd party data
All tests Key: Decoupling A/B test UI from Data processing layer(s)
Analyst manually creates table with list of users ahead of time
A/B system simply allocates from this pool of usersB
RT User participation criteria are defined in UI
Allocate on the fly if a user meets those criteria
Ignite: the winner is ...
Statistical significance + simple highlighting gives “at a glance” read for test cells
Statistical significance values are calculated by the UI so switching the control cell does not require a new query
Druid backend allows interactive query speeds on massive datasets with high cardinality dimensions
UI “falls back” to Presto when the complexity of the query grows beyond simple OLAP
Full stack javascript: Node.js + React
nearly done
start banking some good questions :)
nearly done
a/b testing to∞?
a/b testing to∞?
enabling partners
analyticsself-service tools a/b testing
maximizing value of data for product
? ways to reach me/us @astrobiased ebressert at netflix.com @netflixdata