tensorflow tutorial - university of central...

UNIVERSITY OF CENTRAL FLORIDA

UNIVERSITY OF CENTRAL FLORIDA

TensorFlowTutorial

by Astrid Jackson

UNIVERSITY OF CENTRAL FLORIDA 2

TENSORFLOW

Tensors: n-dimensional arrays Vector: 1-D tensor

Matrix: 2-D tensor

Flow: data flow computation framework A sequence of tensor operations

2


SIMPLE FULLY CONNECTED NETWORK

3

𝒙𝟏

𝒙𝟐

𝒙𝟑

+𝒃𝟏

+𝒃𝟐

+𝒃𝟑

softm

ax𝒚𝟏

𝒚𝟐

𝒚𝟑𝑾𝟑,𝟑

𝑾𝟏,𝟏

𝑾𝟏,𝟑

𝑾𝟐,𝟐

𝑾𝟑,𝟏

𝑾𝟐,𝟑

𝑾𝟑,𝟐

𝑾𝟏,𝟐

𝑾𝟐,𝟏

inputs weights biases outputs


NETWORK AS MATRIX OPERATIONS

4

𝒚𝟏

𝒚𝟐

𝒚𝟑

= softmax

𝑾𝟏,𝟏 𝒙𝟏

𝑾𝟐,𝟏

𝑾𝟑,𝟏

+

𝑾𝟏,𝟐

𝑾𝟐,𝟐

𝑾𝟑,𝟐

𝒙𝟐

𝑾𝟏,𝟑

𝑾𝟐,𝟑

𝑾𝟑,𝟑 𝒙𝟑

𝒃𝟏

𝒃𝟐

𝒃𝟑

∙


NETWORK WITH TENSORFLOW

5

𝒚𝟏

𝒚𝟐

𝒚𝟑

= softmax

𝑾𝟏,𝟏 𝒙𝟏

𝑾𝟐,𝟏

𝑾𝟑,𝟏

+

𝑾𝟏,𝟐

𝑾𝟐,𝟐

𝑾𝟑,𝟐

𝒙𝟐

𝑾𝟏,𝟑

𝑾𝟐,𝟑

𝑾𝟑,𝟑 𝒙𝟑

𝒃𝟏

𝒃𝟐

𝒃𝟑

∙

import tensorflow as tf

logits = tf.matmul(x, w) + b y = tf.nn.softmax(logits)


DEFINE TENSORS

6

𝑾𝟏,𝟏

𝑾𝟐,𝟏

𝑾𝟑,𝟏

𝑾𝟏,𝟐

𝑾𝟐,𝟐

𝑾𝟑,𝟐

𝑾𝟏,𝟑

𝑾𝟐,𝟑

𝑾𝟑,𝟑

w

Variable( initial−value ,name= optional−name )


w = tf.Variable( tf.random_normal([3, 3]), name='w' )

b = tf.Variable( tf.zeros([3]), name='b' )

y = tf.nn.softmax( tf.matmul( x, w ) + b )

𝒃𝟏

𝒃𝟐

𝒃𝟑

b


TENSORFLOW

7





𝑥

MatMul

Add

Variable

VariableCode defines data flow graph

Each variable corresponds to anode in the graph, not the result

Softmax


COMPUTATION AS DATA FLOW GRAPH

8

biases

weights

inputs

targets

MatMul

Add Softmax

Xent

Graph of Nodes, also called operations (ops)


DATA FLOW GRAPH (FORWARD)

9

biases

weights

inputs

targets

MatMul

Add Softmax

Xent

Edges are N-dimensional arrays: Tensors


DATA FLOW GRAPH (BACKWARD GRAPH AND UPDATES)

10

biases

…

learning rate

MulAdd … -=

Backward graph and update are added automatically to graph

'Biases' are variable Some ops compute gradients -= updates biases


TENSORFLOW

11


sess = tf.Session()




print sess.run(y)

𝑥

MatMul

Softmax

Variable

Variable

Code defines data flow graph

• How to execute the graph?

Session

• Manage resource for graph execution

Graph

Fetch

Softmax


INITIALIZE VARIABLE

12


sess = tf.Session()




sess.run( tf.initialize_all_variables() )

print sess.run(y)

Variable is an empty node

• Fill in the content of Variable node

𝑥

MatMul

Softmax

Variable

Variable

Graph

Fetch

Softmax


PLACEHOLDER

13


sess = tf.Session()

x = tf.placeholder( "float", [1, 3] )




sess.run( tf.initialize_all_variables() )

print sess.run(y, feed_dict={x: np.array([[1., 2., 3.]])})

How about x?

placeholder( data_type ,shape= optional_shape ,name= optional_name )

Feed

𝑥

MatMul

Softmax

Variable

Variable

Graph

Fetch

Softmax


SESSION MANAGEMENT

Needs to release resource after use

Using context manager

Interactive session

14

sess.close()

with tf.Session() as sess:…

sess = InteractiveSession()


LOSS

15

labels = tf.placeholder("float", [1, 3])

logits = tf.matmul( x, w ) + b

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(

logits, labels, name='xentropy')

Select loss function

Loss function for softmaxsoftmax_cross_entropy_with_logits(logits, labels,

name= optional−name )


OPTIMIZATION

16



tf.matmul( x, w ) + b, labels, name='xentropy')

optimizer = tf.train.GradientDescentOptimizer(0.1)

train_op = optimizer.minimize(cross_entropy)

sess.run(train_op, feed_dict={x:np.array([[1., 2., 3.]]),

labels:np.aray([[0., 1., 0.]])})

Gradient descent

class GradientDescentOptimizer(learning_rate,use_locking=False,name='GradientDescent')


ITERATIVE UPDATE

17



tf.matmul( x, w ) + b, labels, name='xentropy')

optimizer = tf.train.GradientDescentOptimizer(0.1)

train_op = optimizer.minimize(cross_entropy)

for step in range(10):

sess.run(train_op, feed_dict={x:np.array([[1., 2., 3.]]),

labels:np.array([[0., 1., 0.]])})

Gradient descent requires multiple steps to converge


EVALUATION

18

correct_prediction = tf.equal( tf.argmax(logits, 1),

tf.argmax(labels, 1) )

accuracy = tf.reduce_mean( tf.cast(correct_prediction,

tf.float32) )

print sess.run(accuracy, feed_dict={x:np.array([[1., 2., 3.]]),

labels:np.array([[0., 1., 0.]])})

How well does model do?


ADDING LAYERS

19

x = tf.placeholder("float", [1, 3])

out = x

num_layers = 2

for layer in range(num_layers):

w = tf.Variable( tf.random_normal([3, 3]) )

b = tf.Variable( tf.zeros([1, 3]) )

out = tf.nn.relu( tf.matmul(out, w) + b )

…


GRAPH VISUALIZATION

TensorBoard

Launch TensorBoard

tensorboard --logdir=/tmp/tf_logs

Navigate web browser to:localhost:6006

20

writer = tf.train.SummaryWriter('/tmp/tf_logs', sess.graph)


VISUALIZE STATES

Add summaries

21

summary_op = tf.merge_all_summaries()for step in range(10):

_, summary = sess.run([train_op, summary_op], feed_dict=…)

writer.add_summary(summary, step)

scalar_summary histogram_summary


SAVE AND LOAD MODELS

tf.train.Saver(…)

save(sess, save_path, …)

restore(sess, save_path, …)

22

Default will associate all variables

all_variables()

Replace initialization


ADDING NEW OP

Familiarity with C++ required

Build TensorFlow from source

Eigen Tensorshttps://bitbucket.org/eigen/eigen/src/default/unsupported/Eigen/CXX11/src/Tensor/README.md?fileviewer=file-view-default

23

https://bitbucket.org/eigen/eigen/src/default/unsupported/Eigen/CXX11/src/Tensor/README.md?fileviewer=file-view-default


RESOURCES

http://tensorflow.org

https://github.com/tensorflow/tensorflow

TensorFlow wrappers

Keras: https://keras.io/

Eigen Tensor library


24

http://tensorflow.org/

https://github.com/tensorflow/tensorflow

https://keras.io/


tensorflow tutorial - university of central...

Documents