Download - Large Scale Deep Learning with TensorFlow
![Page 1: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/1.jpg)
Large-Scale Deep LearningWith TensorFlow
Jeff DeanGoogle Brain team
g.co/brainIn collaboration with many other people at Google
![Page 2: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/2.jpg)
We can now store and perform computation on large datasets
MapReduce
![Page 3: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/3.jpg)
We can now store and perform computation on large datasets
But what we really want is not just raw data,but computer systems that understand this data
MapReduce
![Page 4: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/4.jpg)
What do I mean by understanding?
![Page 5: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/5.jpg)
What do I mean by understanding?
![Page 6: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/6.jpg)
What do I mean by understanding?
![Page 7: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/7.jpg)
Neural Networks
![Page 8: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/8.jpg)
“cat”
● A powerful class of machine learning model● Modern reincarnation of artificial neural networks● Collection of simple, trainable mathematical functions
What is Deep Learning?
![Page 9: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/9.jpg)
“cat”
● Loosely based on (what little) we know about the brain
What is Deep Learning?
![Page 10: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/10.jpg)
What is Deep Learning?
● Each neuron is connected to a small subset of other neurons.
● Based on what it sees,it decides what it wants to say.
● Neurons learn to cooperateto accomplish the task.
Commonalities with real brains:
![Page 11: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/11.jpg)
What is Deep Learning?Each neuron implements a relatively simple mathematical function.
But the composition of 106 - 109 such functions is surprisingly powerful.
![Page 12: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/12.jpg)
ConvNets
![Page 13: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/13.jpg)
Important Property of Neural Networks
Results get better with
more data +bigger models +
more computation
(Better algorithms, new insights and improved techniques always help, too!)
![Page 14: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/14.jpg)
Growing Use of Deep Learning at Google
AndroidAppsdrug discoveryGmailImage understandingMapsNatural language understandingPhotosRobotics researchSpeechTranslationYouTube… many others ...
Across many products/areas:
# of directories containing model description files
![Page 15: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/15.jpg)
Open, standard software for general machine learning
Great for Deep Learning in particular
First released Nov 2015
Apache 2.0 license
http://tensorflow.org/and
https://github.com/tensorflow/tensorflow
![Page 16: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/16.jpg)
Strong External Adoption
GitHub Launch Nov. 2015
GitHub Launch Sep. 2013
GitHub Launch Jan. 2012
GitHub Launch Jan. 2008
50,000+ binary installs in 72 hours, 500,000+ since Nov, 2015Most forks of any GitHub repo in 2015, despite only being available starting in Nov, 2015
(source: http://donnemartin.com/viz/pages/2015)
![Page 17: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/17.jpg)
Motivations● DistBelief (our 1st system) was scalable and good for
production deployment, but not as flexible as we wanted for research purposes
● Better understanding of problem space allowed us to make some dramatic simplifications
● Define a standard way of expressing machine learning ideas and computations
● Short circuit the MapReduce/Hadoop inefficiency
![Page 18: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/18.jpg)
http://tensorflow.org/
![Page 19: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/19.jpg)
http://tensorflow.org/
![Page 20: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/20.jpg)
TensorFlow: Expressing High-Level ML Computations
● Core in C++○ Very low overhead
Core TensorFlow Execution System
CPU GPU Android iOS ...
![Page 21: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/21.jpg)
TensorFlow: Expressing High-Level ML Computations
● Core in C++○ Very low overhead
● Different front ends for specifying/driving the computation○ Python and C++ today, easy to add more
Core TensorFlow Execution System
CPU GPU Android iOS ...
![Page 22: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/22.jpg)
TensorFlow: Expressing High-Level ML Computations
● Core in C++○ Very low overhead
● Different front ends for specifying/driving the computation○ Python and C++ today, easy to add more
Core TensorFlow Execution System
CPU GPU Android iOS ...
C++ front end Python front end ...
![Page 23: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/23.jpg)
● Build a graph computing a neural net inference.
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)
x = tf.placeholder("float", shape=[None, 784])
W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
Example TensorFlow fragment
![Page 24: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/24.jpg)
● Automatically add ops to calculate symbolic gradients of variables w.r.t. loss function.
● Apply these gradients with an optimization algorithm
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = -tf.reduce_sum(y_*tf.log(y))
opt = tf.train.GradientDescentOptimizer(0.01)
train_op = opt.minimize(cross_entropy)
Python API for Machine Learning
![Page 25: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/25.jpg)
● Launch the graph and run the training ops in a loop
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
for i in range(1000):
batch_xs, batch_ys = mnist.train.next_batch(100)
sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
Python API for Machine Learning
![Page 26: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/26.jpg)
MatMul
Add Relu
biases
weights
examples
labels
Xent
Graph of Nodes, also called Operations or ops.
Computation is a dataflow graph
![Page 27: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/27.jpg)
with tensors
MatMul
Add Relu
biases
weights
examples
labels
Xent
Edges are N-dimensional arrays: Tensors
Computation is a dataflow graph
![Page 28: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/28.jpg)
with state
Add Mul
biases
...
learning rate
−=...
'Biases' is a variable −= updates biasesSome ops compute gradients
Computation is a dataflow graph
![Page 29: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/29.jpg)
GPU 0 CPU
Add Mul
biases
learning rate
AssignSub
......
distributedComputation is a dataflow graph
![Page 30: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/30.jpg)
GPU 0 CPU
Add Mul
biases
learning rate
AssignSub
......
Assign Devices to Ops● TensorFlow inserts Send/Recv Ops to transport tensors across devices● Recv ops pull data from Send ops
Send Recv
![Page 31: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/31.jpg)
GPU 0 CPU
Add Mul
biases
learning rate
AssignSub
......
Assign Devices to Ops● TensorFlow inserts Send/Recv Ops to transport tensors across devices● Recv ops pull data from Send ops
Send Recv
Send Recv
Send
Recv SendRecv
![Page 32: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/32.jpg)
phones single machines (CPU and/or GPUs) …
distributed systems of 100sof machines and/or GPU cards
Automatically Runs on Variety of Platforms
custom ML hardware
![Page 33: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/33.jpg)
What are some ways thatdeep learning is having
a significant impact at Google?
![Page 34: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/34.jpg)
“How cold is it outside?”
DeepRecurrent
Neural NetworkAcoustic Input Text Output
Reduced word errors by more than 30%
Speech Recognition
Google Research Blog - August 2012, August 2015
![Page 35: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/35.jpg)
“ocean”Deep
ConvolutionalNeural Network
Your Photo
Automatic Tag
Search personal photos without tags.
Google Photos Search
Google Research Blog - June 2013
![Page 36: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/36.jpg)
Google Photos Search
![Page 37: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/37.jpg)
Google Photos Search“Wow. The new Google photo search is a bit insane. I didn’t tag those… :)”
![Page 38: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/38.jpg)
We have tons of vision problems
Image search, StreetView, Satellite Imagery, Translation, Robotics, Self-driving Cars,
www.google.com/sunroof
![Page 39: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/39.jpg)
“Seeing” Go
![Page 40: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/40.jpg)
April 1, 2009: April Fool’s Day joke
Nov 5, 2015: Launched Real Product
Feb 1, 2016: >10% of mobile Inbox replies
Smart Reply
![Page 41: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/41.jpg)
Small Feed-Forward
Neural Network
Incoming Email
ActivateSmart Reply?
yes/no
Smart Reply Google Research Blog- Nov 2015
![Page 42: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/42.jpg)
Small Feed-Forward
Neural Network
Incoming Email
ActivateSmart Reply?
Deep RecurrentNeural Network
Generated Replies
yes/no
Smart Reply Google Research Blog- Nov 2015
![Page 43: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/43.jpg)
Combined Vision + Translation
![Page 44: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/44.jpg)
Model: A close up of a child holding a stuffed animal.
Human: A young girl asleep on the sofa cuddling a stuffed bear.
Model: A baby is asleep next to a teddy bear.
Image Captions Research
![Page 45: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/45.jpg)
![Page 46: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/46.jpg)
Experiment Turnaround Time and Research Productivity
● Minutes, Hours:○ Interactive research! Instant gratification!
● 1-4 days○ Tolerable○ Interactivity replaced by running many experiments in parallel
● 1-4 weeks○ High value experiments only○ Progress stalls
● >1 month○ Don’t even try
![Page 47: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/47.jpg)
![Page 48: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/48.jpg)
Image Model Training Time
Hours
10 GPUs50 GPUs
1 GPU
![Page 49: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/49.jpg)
Hours
2.6 hours vs. 79.3 hours (30.5X)
10 GPUs50 GPUs
1 GPU
Image Model Training Time
![Page 50: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/50.jpg)
How Can You Get Started with Machine Learning?Four ways, with varying complexity:
(1) Use a Cloud-based API (Vision, Speech, etc.)(2) Run your own pretrained model(3) Use an existing model architecture, and
retrain it or fine tune on your dataset(4) Develop your own machine learning models
for new problems
More flexible,
but more effort
required
![Page 51: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/51.jpg)
(1) Use Cloud-based APIs
cloud.google.com/translate
cloud.google.com/speech
cloud.google.com/vision
cloud.google.com/text
![Page 52: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/52.jpg)
(1) Use Cloud-based APIs
cloud.google.com/translate
cloud.google.com/speech
cloud.google.com/vision
cloud.google.com/text
![Page 53: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/53.jpg)
Google Cloud Vision APIhttps://cloud.google.com/vision/
![Page 54: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/54.jpg)
(2) Using a Pre-trained Image Model yourself with TensorFlowwww.tensorflow.org/tutorials/image_recognition/index.html
![Page 55: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/55.jpg)
For training your own models (3 & 4), two choices:
cloud.google.com/ml
or
Run open-source release on your own physical machines or virtual machines in a cloud hosting environment
![Page 56: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/56.jpg)
(3) Training a Model on Your Own Image Datawww.tensorflow.org/versions/master/how_tos/image_retraining/index.html
![Page 57: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/57.jpg)
(4) Develop your own machine learning modelshttps://www.tensorflow.org/versions/master/get_started/basic_usage.html
![Page 58: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/58.jpg)
Example: Combining Vision with Robotics
“Deep Learning for Robots: Learning from Large-Scale Interaction”, Google Research Blog, March, 2016
“Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection”, Sergey Levine, Peter Pastor, Alex Krizhevsky, & Deirdre Quillen,Arxiv, arxiv.org/abs/1603.02199
https://www.youtube.com/watch?v=iaF43Ze1oeI
![Page 59: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/59.jpg)
What Does the Future Hold?Deep learning usage will continue to grow and accelerate:
● Across more and more fields and problems:○ robotics, self-driving vehicles, ...○ health care○ video understanding○ dialogue systems○ personal assistance○ ...
![Page 60: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/60.jpg)
ConclusionsDeep neural networks are making significant strides in understanding:In speech, vision, language, search, …
If you’re not considering how to use deep neural nets to solve your vision or understanding problems, you almost certainly should be
![Page 61: Large Scale Deep Learning with TensorFlow](https://reader034.vdocuments.net/reader034/viewer/2022052318/58f9a900760da3da068b69de/html5/thumbnails/61.jpg)
Further Reading● Dean, et al., Large Scale Distributed Deep Networks, NIPS 2012, research.google.
com/archive/large_deep_networks_nips2012.html. ● TensorFlow white paper, tensorflow.org/whitepaper2015.pdf (clickable links in bibliography)● TensorFlow: A System for Large-Scale Machine Learning, http://arxiv.org/abs/1605.08695
tensorflow.org
research.google.com/people/jeff
We’re Hiring! See g.co/brain
Questions?