Introduction toPresented by Xie Yaochen

2017.05.13

A brief introduction to Deep Learning

Get started with a Neural Networks

Advanced Models

Convolutional Neural Networks (CNN)

Advanced Models

Convolutional Neural Networks (CNN)

Convolutional Layers

Advanced Models

Convolutional Neural Networks (CNN)

Pooling Layers

Advanced Models

Recurrent Neural Networks (RNN)

Advanced Models

Recurrent Neural Networks (RNN)

Variants of RNN:

LSTM

GRU

Advanced Models

Methods/Tricks to deal with Overfitting

Regularization

Activation (Relu…)

Dropout

Batch & Batch normalization

….

Advanced Models

Optimizer

Stochastic gradient descent (SGD)

Momentum

Adagrad

Adam

….

What is

TensorFlow™ is an open source software library for numerical computation using data flow graphs open-sourced by Google.

But what does it actually do? TensorFlow provides primitives for defining functions on tensors and

automatically computing their derivatives.

Okay, I know. So what is a Tensor?

Tensor: N-dimensional array

A scalar is a tensor

A vector is a tensor

A matrix is a tensor

e.g. Image represented as 3-d tensor rows, cols, channels(RGB)

3 # a rank 0 tensor; this is a scalar with shape []

[1. ,2., 3.] # a rank 1 tensor; this is a vector with shape [3]

[[1., 2., 3.], [4., 5., 6.]] # a rank 2 tensor; a matrix with shape [2, 3]

[[[1., 2., 3.]], [[7., 8., 9.]]] # a rank 3 tensor with shape [2, 1, 3]

When and Why should we use TensorFlow?

Frame work

Developing Language

Supported API 安装难度 灵活性 上⼿手难度

Caffe C++/CUDA C++/python/ matlab

*** ** **

mxNet C++/CUDA Matlab/JS/ C++/Scala

** ** *

Tensorflow C++/CUDA/ python

C++/python * *** ***

Other: Theano, Torch… (CPU only)

How to use

One command to install TensorFlow

$ pip install tensorflow

( for Linux / Mac OS )

or install by Anaconda

( for Windows)

Two steps to run your TensorFlow

Building the computational graph

Running the computational graph

Re : Zero 从零开始的Tensorflow

(假设⼤大家都会python以及常⽤用库，如numpy，的基本使⽤用⽅方法)

Define a constant:

node1 = tf.constant(3.0, tf.float32)

node2 = tf.constant(4.0) # also tf.float32 implicitly

print(node1, node2)

Output

Tensor("Const:0", shape=(), dtype=float32) Tensor("Const_1:0", shape=(), dtype=float32)

Re : Zero 从零开始的Tensorflow

Get value of a tensor

sess = tf.Session()

print(sess.run([node1, node2]))

Output

[3.0, 4.0]

Re : Zero 从零开始的Tensorflow

Operations

node3 = tf.add(node1, node2)

print("node3: ", node3)

print("sess.run(node3): ",sess.run(node3))

Output

node3: Tensor("Add_2:0", shape=(), dtype=float32)

sess.run(node3): 7.0

Re : Zero 从零开始的Tensorflow

Placeholder

a = tf.placeholder(tf.float32)

b = tf.placeholder(tf.float32)

adder_node = a + b # + provides a shortcut for tf.add(a, b)

print(sess.run(adder_node, {a: 3, b:4.5}))

print(sess.run(adder_node, {a: [1,3], b: [2, 4]}))

Output

7.5

[ 3. 7.]

Re : Zero 从零开始的Tensorflow

Variable

W = tf.Variable([.3], tf.float32)

b = tf.Variable([-.3], tf.float32)

x = tf.placeholder(tf.float32)

linear_model = W * x + b

Initialize and run session

init = tf.global_variables_initializer()

sess.run(init)

Re : Zero 从零开始的Tensorflow

Variable

print(sess.run(linear_model, {x:[1,2,3,4]}))

Output

[ 0. 0.30000001 0.60000002 0.90000004]

Re : Zero 从零开始的Tensorflow

To evaluate the model

y = tf.placeholder(tf.float32)

squared_deltas = tf.square(linear_model - y)

loss = tf.reduce_sum(squared_deltas)

print(sess.run(loss, {x:[1,2,3,4], y:[0,-1,-2,-3]}))

Output

23.66

Re : Zero 从零开始的Tensorflow

Optimizing

optimizer = tf.train.GradientDescentOptimizer(0.01)

train = optimizer.minimize(loss)

sess.run(init) # reset values to incorrect defaults.

for i in range(1000):

sess.run(train, {x:[1,2,3,4], y:[0,-1,-2,-3]})

print(sess.run([W, b]))

MNIST - To Say “Hello World!”

MNIST is a simple computer vision dataset. It consists of images of handwritten digits like these:

Each image is 28 pixels by 28 pixels. We can interpret this as a big array of numbers:

MNIST - To Say “Hello World!”

Step 1: Pretreat the images and labels (total size = 55000)

Flatten this array into a vector of 28x28 = 784 numbers

Convert the labels into one-hot vectors

(For example, 3 would be [0,0,0,1,0,0,0,0,0,0] )

MNIST - To Say “Hello World!”

Step 2: Softmax Regressions

MNIST - To Say “Hello World!”

Step 3: To train and evaluate the model

Loss : Cross-entropy

Where y is our predicted probability distribution, and y′ is the true distribution (the one-hot vector with the digit labels)

train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

MNIST - A Further Example (CNN)

Convolution and Pooling (see example3.py)

References:

1. The MNIST Database: http://yann.lecun.com/exdb/mnist/

2. TensorFlow Official Document: https://www.tensorflow.org/

3. Colah’s Blog: http://colah.github.io/

4. Stanford Course CS224d: https://cs224d.stanford.edu/lectures/

5. Prof. Jordi Torres’ Home Page: http://www.jorditorres.org/

6. Fabien Baradel’s Blog: https://fabienbaradel.github.io/

Also where you could learn more about Deep Learning and TensorFlow