tensorflow tutorial - university of central...

UNIVERSITY OF CENTRAL FLORIDA

UNIVERSITY OF CENTRAL FLORIDA

TensorFlowTutorial

by Astrid Jackson

UNIVERSITY OF CENTRAL FLORIDA 2

TENSORFLOW

Tensors: n-dimensional arrays Vector: 1-D tensor

Matrix: 2-D tensor

Flow: data flow computation framework A sequence of tensor operations

2

UNIVERSITY OF CENTRAL FLORIDA 3

SIMPLE FULLY CONNECTED NETWORK

3

𝒙𝟏

𝒙𝟐

𝒙𝟑

+𝒃𝟏

+𝒃𝟐

+𝒃𝟑

softm

ax𝒚𝟏

𝒚𝟐

𝒚𝟑𝑾𝟑,𝟑

𝑾𝟏,𝟏

𝑾𝟏,𝟑

𝑾𝟐,𝟐

𝑾𝟑,𝟏

𝑾𝟐,𝟑

𝑾𝟑,𝟐

𝑾𝟏,𝟐

𝑾𝟐,𝟏

inputs weights biases outputs

UNIVERSITY OF CENTRAL FLORIDA 4

NETWORK AS MATRIX OPERATIONS

4

𝒚𝟏

𝒚𝟐

𝒚𝟑

= softmax

𝑾𝟏,𝟏 𝒙𝟏

𝑾𝟐,𝟏

𝑾𝟑,𝟏

+

𝑾𝟏,𝟐

𝑾𝟐,𝟐

𝑾𝟑,𝟐

𝒙𝟐

𝑾𝟏,𝟑

𝑾𝟐,𝟑

𝑾𝟑,𝟑 𝒙𝟑

𝒃𝟏

𝒃𝟐

𝒃𝟑

∙

UNIVERSITY OF CENTRAL FLORIDA 5

NETWORK WITH TENSORFLOW

5

𝒚𝟏

𝒚𝟐

𝒚𝟑

= softmax

𝑾𝟏,𝟏 𝒙𝟏

𝑾𝟐,𝟏

𝑾𝟑,𝟏

+

𝑾𝟏,𝟐

𝑾𝟐,𝟐

𝑾𝟑,𝟐

𝒙𝟐

𝑾𝟏,𝟑

𝑾𝟐,𝟑

𝑾𝟑,𝟑 𝒙𝟑

𝒃𝟏

𝒃𝟐

𝒃𝟑

∙

import tensorflow as tf

logits = tf.matmul(x, w) + b y = tf.nn.softmax(logits)

UNIVERSITY OF CENTRAL FLORIDA 6

DEFINE TENSORS

6

𝑾𝟏,𝟏

𝑾𝟐,𝟏

𝑾𝟑,𝟏

𝑾𝟏,𝟐

𝑾𝟐,𝟐

𝑾𝟑,𝟐

𝑾𝟏,𝟑

𝑾𝟐,𝟑

𝑾𝟑,𝟑

w

Variable( initial−value ,name= optional−name )

import tensorflow as tf

w = tf.Variable( tf.random_normal([3, 3]), name='w' )

b = tf.Variable( tf.zeros([3]), name='b' )

y = tf.nn.softmax( tf.matmul( x, w ) + b )

𝒃𝟏

𝒃𝟐

𝒃𝟑

b

UNIVERSITY OF CENTRAL FLORIDA 7

TENSORFLOW

7

import tensorflow as tf

w = tf.Variable( tf.random_normal([3, 3]), name='w' )

b = tf.Variable( tf.zeros([3]), name='b' )

y = tf.nn.softmax( tf.matmul( x, w ) + b )

𝑥

MatMul

Add

Variable

VariableCode defines data flow graph

Each variable corresponds to anode in the graph, not the result

Softmax

UNIVERSITY OF CENTRAL FLORIDA 8

COMPUTATION AS DATA FLOW GRAPH

8

biases

weights

inputs

targets

MatMul

Add Softmax

Xent

Graph of Nodes, also called operations (ops)

UNIVERSITY OF CENTRAL FLORIDA 9

DATA FLOW GRAPH (FORWARD)

9

biases

weights

inputs

targets

MatMul

Add Softmax

Xent

Edges are N-dimensional arrays: Tensors

UNIVERSITY OF CENTRAL FLORIDA 10

DATA FLOW GRAPH (BACKWARD GRAPH AND UPDATES)

10

biases

…

learning rate

MulAdd … -=

Backward graph and update are added automatically to graph

'Biases' are variable Some ops compute gradients -= updates biases

UNIVERSITY OF CENTRAL FLORIDA 11

TENSORFLOW

11

import tensorflow as tf

sess = tf.Session()

w = tf.Variable( tf.random_normal([3, 3]), name='w' )

b = tf.Variable( tf.zeros([3]), name='b' )

y = tf.nn.softmax( tf.matmul( x, w ) + b )

print sess.run(y)

𝑥

MatMul

Softmax

Variable

Variable

Code defines data flow graph

• How to execute the graph?

Session

• Manage resource for graph execution

Graph

Fetch

Softmax

UNIVERSITY OF CENTRAL FLORIDA 12

INITIALIZE VARIABLE

12

import tensorflow as tf

sess = tf.Session()

w = tf.Variable( tf.random_normal([3, 3]), name='w' )

b = tf.Variable( tf.zeros([3]), name='b' )

y = tf.nn.softmax( tf.matmul( x, w ) + b )

sess.run( tf.initialize_all_variables() )

print sess.run(y)

Variable is an empty node

• Fill in the content of Variable node

𝑥

MatMul

Softmax

Variable

Variable

Graph

Fetch

Softmax

UNIVERSITY OF CENTRAL FLORIDA 13

PLACEHOLDER

13

import tensorflow as tf

sess = tf.Session()

x = tf.placeholder( "float", [1, 3] )

w = tf.Variable( tf.random_normal([3, 3]), name='w' )

b = tf.Variable( tf.zeros([3]), name='b' )

y = tf.nn.softmax( tf.matmul( x, w ) + b )

sess.run( tf.initialize_all_variables() )

print sess.run(y, feed_dict={x: np.array([[1., 2., 3.]])})

How about x?

placeholder( data_type ,shape= optional_shape ,name= optional_name )

Feed

𝑥

MatMul

Softmax

Variable

Variable

Graph

Fetch

Softmax

UNIVERSITY OF CENTRAL FLORIDA 14

SESSION MANAGEMENT

Needs to release resource after use

Using context manager

Interactive session

14

sess.close()

with tf.Session() as sess:…

sess = InteractiveSession()

UNIVERSITY OF CENTRAL FLORIDA 15

LOSS

15

labels = tf.placeholder("float", [1, 3])

logits = tf.matmul( x, w ) + b

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(

logits, labels, name='xentropy')

Select loss function

Loss function for softmaxsoftmax_cross_entropy_with_logits(logits, labels,

name= optional−name )

UNIVERSITY OF CENTRAL FLORIDA 16

OPTIMIZATION

16

labels = tf.placeholder("float", [1, 3])

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(

tf.matmul( x, w ) + b, labels, name='xentropy')

optimizer = tf.train.GradientDescentOptimizer(0.1)

train_op = optimizer.minimize(cross_entropy)

sess.run(train_op, feed_dict={x:np.array([[1., 2., 3.]]),

labels:np.aray([[0., 1., 0.]])})

Gradient descent

class GradientDescentOptimizer(learning_rate,use_locking=False,name='GradientDescent')

UNIVERSITY OF CENTRAL FLORIDA 17

ITERATIVE UPDATE

17

labels = tf.placeholder("float", [1, 3])

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(

tf.matmul( x, w ) + b, labels, name='xentropy')

optimizer = tf.train.GradientDescentOptimizer(0.1)

train_op = optimizer.minimize(cross_entropy)

for step in range(10):

sess.run(train_op, feed_dict={x:np.array([[1., 2., 3.]]),

labels:np.array([[0., 1., 0.]])})

Gradient descent requires multiple steps to converge

UNIVERSITY OF CENTRAL FLORIDA 18

EVALUATION

18

correct_prediction = tf.equal( tf.argmax(logits, 1),

tf.argmax(labels, 1) )

accuracy = tf.reduce_mean( tf.cast(correct_prediction,

tf.float32) )

print sess.run(accuracy, feed_dict={x:np.array([[1., 2., 3.]]),

labels:np.array([[0., 1., 0.]])})

How well does model do?

UNIVERSITY OF CENTRAL FLORIDA 19

ADDING LAYERS

19

x = tf.placeholder("float", [1, 3])

out = x

num_layers = 2

for layer in range(num_layers):

w = tf.Variable( tf.random_normal([3, 3]) )

b = tf.Variable( tf.zeros([1, 3]) )

out = tf.nn.relu( tf.matmul(out, w) + b )

…

UNIVERSITY OF CENTRAL FLORIDA 20

GRAPH VISUALIZATION

TensorBoard

Launch TensorBoard

tensorboard --logdir=/tmp/tf_logs

Navigate web browser to:localhost:6006

20

writer = tf.train.SummaryWriter('/tmp/tf_logs', sess.graph)

UNIVERSITY OF CENTRAL FLORIDA 21

VISUALIZE STATES

Add summaries

21

summary_op = tf.merge_all_summaries()for step in range(10):

_, summary = sess.run([train_op, summary_op], feed_dict=…)

writer.add_summary(summary, step)

scalar_summary histogram_summary

UNIVERSITY OF CENTRAL FLORIDA 22

SAVE AND LOAD MODELS

tf.train.Saver(…)

save(sess, save_path, …)

restore(sess, save_path, …)

22

Default will associate all variables

all_variables()

Replace initialization

UNIVERSITY OF CENTRAL FLORIDA 23

ADDING NEW OP

Familiarity with C++ required

Build TensorFlow from source

Eigen Tensorshttps://bitbucket.org/eigen/eigen/src/default/unsupported/Eigen/CXX11/src/Tensor/README.md?fileviewer=file-view-default

23

UNIVERSITY OF CENTRAL FLORIDA 24

RESOURCES

http://tensorflow.org

https://github.com/tensorflow/tensorflow

TensorFlow wrappers

Keras: https://keras.io/

Eigen Tensor library

https://bitbucket.org/eigen/eigen/src/default/unsupported/Eigen/CXX11/src/Tensor/README.md?fileviewer=file-view-default

24