how gpus power comcast's x1 voice remote and smart video ...€¦ · how gpus power...
TRANSCRIPT
![Page 1: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/1.jpg)
How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics
Jan NeumannComcast Labs DCMay 10th, 2017
![Page 2: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/2.jpg)
2
Comcast Applied Artificial Intelligence Lab
SmartHome
SmartTV
SmartInternet
Media&
VideoAnalytics
DeepLearning
DataScience
Recommendations
&Search
Voice&
NLP
![Page 3: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/3.jpg)
3
Today: How Comcast Uses AI to Evolve and Reinvent the TV Experience
Smart Home
SmartTV
SmartInternet
Media&
VideoAnalytics
DeepLearning
DataScience
Recommendations
&Search
Voice&
NLP
![Page 4: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/4.jpg)
4
Netflix
LIVETV
OnlineVideo
AI for Content Discovery –Voice Search
![Page 5: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/5.jpg)
5
• Query: “HBO”
X1 Smart TV with Voice
AnswerSelector
Voice remote ASR
query
NLP modules
action
Set-top Box TV
![Page 6: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/6.jpg)
6
Open NLP: Multiple Domains with Voice
TV
HOME
.
.
.
queryDomain
Selector
Answer
Selector
.
.
.
Answer
Selectorresponse
CUSTOMERCARE
NEWS
![Page 7: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/7.jpg)
7
Open NLP: Multiple Domains with Voice
TV
HOMEturnontheheat
Domain
Selector
Answer
SelectorAnswer
Selector
CUSTOMERCARE
NEWS
response
0.80
0.15
0.02
0.03
Selected={TV,Home} Precision=100%
Applicable={TV,Home} Recall=100%
Threshold=0.10
![Page 8: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/8.jpg)
8
Open NLP: Multiple Domains with Voice
TV
HOME
Showmemypassword
Domain
Selector
Answer
SelectorAnswer
Selector
CUSTOMERCARE
NEWS
response
0.03
0.04
0.03
0.90
Selected={CustomerCare} Precision=100%
Applicable={CustomerCare} Recall=100%
Threshold=0.10
![Page 9: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/9.jpg)
9
• Cascade of Deep Learning Models of increasing complexity
Domain Selector in Practice
Entity Detection
Service“HBO”
SimpleModel
ComplexModel
SENDTODOMAIN
DONOTSENDTODOMAIN
YES
YES
NO YES
NO
NO
![Page 10: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/10.jpg)
10
SENDTODOMAIN
NO
• Cascade of Deep Learning Models of increasing complexity
Domain Selector in Practice
Entity Detection
Service
“Show me funny
comedies”
SimpleModel
ComplexModel
DONOTSENDTODOMAIN
YES
YES
NO YES
NO
![Page 11: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/11.jpg)
11
• Query: “who plays the oracle in matrix”
X1 Smart TV with Voice
Voice remote
query action
QA Answer (id or text)
Question(text)
ASR NLP modules Set-top Box TV
![Page 12: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/12.jpg)
12
• Given:• Question in natural-language form q• Structured knowledge base that contains list of facts• [ subject – relation – (attribute) – object ]
• Return: • Answer to q
• Assuming:• q answerable by a single fact.• Source entity mentioned in q.• Answer is neighbor of source entity node.
First-order Question Answering
subject object
attribute
“Matrix” “Keanu Reeves”
“Neo”
“Tom Hanks” “9/1/1956”
![Page 13: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/13.jpg)
13
Question Answering with Knowledge Graph
Predict
Relation
Question
Extract
Entities [e1,…,eN ]names/titles
Structured
Query
Subj=e1Obj=?Rel=r
Knowledge
Graph
Search
e1 |r |e2
relation
r
Generate
Answer
TextanswerTrain
subj |rel |obj
Howoldis
TomHanks?
![Page 14: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/14.jpg)
14
Question Answering with Knowledge Graph
Predict
Relation
Question
Extract
Entities [e1,…,eN ]names/titles
Structured
Query
Subj=e1Obj=e2Rel=r
Knowledge
Graph
Search
e1 |r |e2
relation
r
Generate
Answer
TextanswerTrain
subj |rel |obj
[e1,…,eN ]names/titles
Subj=TomHanksRel=birthObj =?relation
r
TomHanks
is55years
old.
birth
Tom
Hanks TomHanks|birth|1956
TomHanks
is59yearsold
Howoldis
TomHanks?
![Page 15: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/15.jpg)
15
EntityDetection [ e1, …, eN ]
names / titles
PredictRelation
relationr
subj=eobj=?attr=?rel=r
Question Answering with Knowledge Graph using Recurrent Neural Networks (RNNs)
StructuredQuery
Question
where Tom Hanks was
placeof birth
born
memory
where Tom Hanks was
NA Subj Subj NA NA
born
mem
ory Entity Detection ~ Tagging Relation Prediction ~ Classification
![Page 16: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/16.jpg)
16
word
hidden
input
output
0.39 0.61
washingtonheights
0.89 0.11
memory
Recurrent Neural Networks
LOC PER PERLOC
![Page 17: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/17.jpg)
17
Netflix
LIVETV
OnlineVideo
AI for Content Discovery – Automatic Content Analysis
![Page 18: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/18.jpg)
18
Most metadata is at the asset level
• Genres• Credits• Synopsis• Keywords
![Page 19: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/19.jpg)
19
Much more data exists within the asset
• Chapters• Moments• Annotations
MovieFrameShotScene
Chapter
![Page 20: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/20.jpg)
20
Why is this useful?
Whoisinthis
scene?
Whatarethebest
momentsonTV?
In-game
highlight
navigation
Search&
Recommendations
![Page 21: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/21.jpg)
21
How does Automatic Content Analysis work?
ComputerVision
Audio Analysis
NaturalLanguage
Processing
AI &Machine Learning
Chaptering
Scene-levelAnnotations
Video
Frame-levelAnnotations
![Page 22: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/22.jpg)
22
Why is it possible now?
Large-scale Image recognition performance
Big
Data
Better
Algorithms
(Deeplearning)
Cloud/GPU
Computing
![Page 23: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/23.jpg)
23
Super-human accuracy in speech and image recognition!
Large-scale Image recognition performance
Big
Data
Better
Algorithms
(Deeplearning)
Cloud/GPU
Computing
![Page 24: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/24.jpg)
24
New experiences!
Big
Data
Better
Algorithms
(Deeplearning)
Cloud/GPU
Computing
![Page 25: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/25.jpg)
25
• Place highlights over games recorded onto customers’ DVRs for football, baseball, hockey, basketball and soccer.
Example Application: In-Game Highlights
“I’llrecordasmanygamesasIcan.WhenIdon’twanttowatchthewholegame,it’sagreatwaytodoit.”– CustomerTestimonial
“In-GameHighlights”
FeatureforNFLhasbeen
releasedonComcastX1
lastfall
![Page 26: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/26.jpg)
26
Netflix
LIVETV
OnlineVideo
AI for Content Discovery – Personalization
![Page 27: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/27.jpg)
27
+
=
Personalized Entertainment Experiences
What is popular right now? What do you like?
PersonalizedRecommendations
![Page 28: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/28.jpg)
28
Deep learning-based recommender system for Live TV - Training a joint embedding space to combine the scores- Channel- and Program-based recommendations
- Time-dependent recommendations
- Trending/popular and personal favorite channels, programs, sport teams
- Rich content descriptions from automatic content analysis
What should I watch right now?
Live TVRecommenderSystem
Favorite
Channels
Favorite
Programs
Collaborative
Filtering
Trending
Popularity
Content
Descriptions
![Page 29: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/29.jpg)
29
Netflix
LIVETV
OnlineVideo
Deep Learning Infrastructure
![Page 30: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/30.jpg)
30
• Deep Learning Frameworks – Keras, Tensorflow, Theano, PyTorch, Caffee (older models)
• All deployments using nvidia-docker– Thanks to Nvidia solutions team to help with best practices
• All deep learning training done on multi-GPU servers– NvidiaTesla (Production) and 8xTitan X (Dev) GPUs– Nvidia DGX-1 for large scale training – video and nlp
• Next steps– Container scheduler – Kubernetes and Hashicorp Nomad– Network compression/simplification for increased efficiency (TensorRT)
Deep Learning Infrastructure
![Page 31: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/31.jpg)
31
Machine Learning Data Science
Big Data AI
Improving Customer
Experience Everywhere at
Comcast/NBCU
Deep Learning-based ML is applied everywhere at Comcast
HighSpeedInternet
Video
IPTelephony
HomeSecurity/
Automation
UniversalParks
MediaProperties
Formoreinfosee:
dclabs.comcast.com
![Page 32: How GPUs Power Comcast's X1 Voice Remote and Smart Video ...€¦ · How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics Jan Neumann Comcast Labs DC May 10th, 2017](https://reader034.vdocuments.net/reader034/viewer/2022051918/600abaf0e345a42d56409cff/html5/thumbnails/32.jpg)