quoc le, software engineer, google at mlconf sf - 11/13/15
TRANSCRIPT
![Page 1: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/1.jpg)
Quoc V. Le
Deep Learning: Overview and Latest Results
joint work with many Google & Stanford collaborators
![Page 3: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/3.jpg)
Coates et al 2009
![Page 4: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/4.jpg)
Quoc V. Le
Feature Engineering
Handcrafted Features
Classifier
X X’
“Face”Y
![Page 5: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/5.jpg)
Quoc V. Le
Why Deep Learning?
x
Corner Detectors
…
Edge Detectors
Raw Image
Cat, Face Detectors
![Page 6: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/6.jpg)
Quoc V. Le
Model Partitioning
Machine 4
Machine 1 Machine 2
Machine 3
![Page 7: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/7.jpg)
Quoc V. Le
GoogleBrain infrastructure
ParameterServer
ModelWorkers
DataShards
p’ = p + ∆p
∆p ∆p ∆p
Le et al, 2011, Dean et al 2011
![Page 8: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/8.jpg)
Quoc V. Le
Why scaling up?
Amount of data
Accuracy
Deep Learning
Traditional Machine Learning
![Page 9: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/9.jpg)
Quoc V. Le
GoogleBrain teaches itself
Watch YouTube for 3 days (10M frames)
Run on 16,000 machines1 billion synapses
Le et al 2011
![Page 10: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/10.jpg)
Quoc V. Le
GoogleBrain teaches itself
Watch YouTube for 3 days (10M frames)
Run on 16,000 machines1 billion synapses
“Cat” Le et al 2011
![Page 11: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/11.jpg)
Quoc V. Le
![Page 12: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/12.jpg)
Quoc V. Le
Applications
- Understanding Images
- Understanding Speech
- Understanding Texts
![Page 13: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/13.jpg)
Quoc V. Le
Image Recognition
28% vs. 5%Error rate:
Achieves superhuman performance
Now used in Google+ Photosearch and StreetView
LeCun et al 1998, Krizhevsky et al 2012
![Page 14: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/14.jpg)
Quoc V. Le
Speech Recognition
Voice Search
23% vs. 8%Error rate:
Hinton et al, 2011
![Page 15: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/15.jpg)
Quoc V. Le
Applications
- Understanding Images
- Understanding Speech
- Understanding Texts
![Page 16: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/16.jpg)
Quoc V. Le
Mapping sequence to sequence
A
W X Y Z <EOS>
B C <EOS> W X Y
W
Z
WX XWY YXWZ YXW<EOS>
![Page 17: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/17.jpg)
Quoc V. Le
Sample ConversationsDescribe your problem: i am having issues accessingvpn
Machine: hiHuman: helloMachine: could you please let me know what are the operating systems you are using ?Human: i am using linuxMachine: ok it was in the ssl vpn on which you stated it was brokenMachine: please check if you are logged in to corp networkMachine: what is the error that you are running pleaseHuman: i am seeing an error related to vpnMachine: what is the error message that you are getting when connecting to vpn using network connect ?Human: connection refused or something like thatMachine: may i know the version of network connect you connect ?
![Page 18: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/18.jpg)
Quoc V. Le
Sample ConversationsHuman: i am not sure i know thatMachine: is the network connect prompting for any user certificateMachine: ?Human: yesMachine: are you at home or at work at the moment ?Machine: office ?Human: i am at homeMachine: try thisMachine: goto <URL> page and enter your name and passwordHuman: doneMachine: try logging in with <NAME> and check if its working fine nowHuman: yes , now it works !Machine: greatMachine: anything else that i can helpHuman: no , that was very usefulMachine: you are other than <NAME> : )Machine: have a nice night
![Page 19: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/19.jpg)
Quoc V. Le
Sample Conversations
Human: what is the purpose of life ?Machine: to serve the greater good .Human: what is the purpose of living ?Machine: to live forever .Human: what is the purpose of existence ?Machine: to find out what happens when we get to the planet earth .Human: where are you now ?Machine: i ’m in the middle of nowhere .Human: what is the purpose of dying ?Machine: to have a life .Human: what is the purpose of being intelligent ?Machine: to find out what it is .Human: what is the purpose of emotions ?Machine: i don ’t know .
![Page 20: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/20.jpg)
Quoc V. Le
Applications
- Understanding Images
- Understanding Speech
- Understanding TextsImage to Texts
![Page 21: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/21.jpg)
Quoc V. Le
Generating captions for images
A person riding a motorcycle on a dirt road
A group of young people playing a game of frisbee
Two dogs play in the grass
Vinyals, et al, 2015
![Page 22: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/22.jpg)
Quoc V. Le
Deep Learning tutorials & software
Deep Learning tutorials are available on my website:
Part 1 (Basic): http://ai.stanford.edu/~quocle/tutorial1.pdf
Part 2 (Advanced): http://ai.stanford.edu/~quocle/tutorial2.pdf
Popular open source software for Deep Learning: CAFFE https://github.com/BLVC/caffe
![Page 23: Quoc Le, Software Engineer, Google at MLconf SF - 11/13/15](https://reader036.vdocuments.net/reader036/viewer/2022062902/58ef57cf1a28ab8e388b4567/html5/thumbnails/23.jpg)
Quoc V. Le
Thanks
Jeff Dean Oriol VinyalsIlya SutskeverGreg Corrado
Matthieu DevinAndrew Ng Rajat MongaNavdeep Jaitly
+GoogleBrain team, Tomas Mikolov, Thang Luong, William Chan, Wojciech Zaremba