Upload File

recurrent neural networksbhiksha/courses/deeplearning/...the backpropagation through time (btt)...

  • Home
  • Documents
  • Recurrent Neural Networksbhiksha/courses/deeplearning/...The Backpropagation Through Time (BTT) Algorithm Different Recurrent Neural Network (RNN) paradigms How Layering RNNs works
25
Deep Learning - Recitation (2/16/2018) Recurrent Neural Networks

Upload: others

Post on 12-Feb-2021

8 views

Category:

Documents


0 download

Report

TRANSCRIPT

  • Deep Learning - Recitation (2/16/2018)

    Recurrent Neural Networks

  • ● What is an RNN

    ● The Backpropagation Through Time (BTT) Algorithm

    ● Different Recurrent Neural Network (RNN) paradigms

    ● How Layering RNNs works

    ● Popular Types of RNN Cells

    ● Common Pitfalls of RNNs

    Table of Contents

  • An RNN is a type of artificial neural network in where the weights form a directed cycle

    What is an RNN

    Let’s take a step back to a typical feedforward NN to explain what this means…

  • An RNN is a type of artificial neural network in where the weights form a directed cycle

    What is an RNN

    Let’s say this is 0

    And this is 0

  • An RNN is a type of artificial neural network in where the weights form a directed cycle

    What is an RNN

  • Backpropagation Through Time

    Note: This is handwavy

    We can already back propagate error through this...

  • Backpropagation Through Time

    Note: This is handwavy

    What does this look like?

  • Backpropagation Through Time

    Note: This is handwavy

    What if we rearrange some things?

    Note that we dropped Hidden0

    Note that we dropped Hidden2

    This just looks like a feedforward network with some strange connections...

  • Backpropagation Through Time

    Note: This is handwavy

    These two can be trained in exactly the same way!

    Regular Backprop!

  • Backpropagation Through Time

    Note: This is handwavy

    Calculate the gradients for all of these weights

    “Unroll” your network

  • Backpropagation Through Time

    Note: This is handwavy

    Because all these weights are tied update them at the same time… Just like tied weights in a CNN

    “Unroll” your network

  • RNN Paradigms

    Different problems are more suited for different RNN paradigms

  • RNN Paradigms

    Different problems are more suited for different RNN paradigms

    Given a single input predict a single output

    This is just a simple feedforward neural network

  • RNN Paradigms

    Different problems are more suited for different RNN paradigms

    Given a single input predict a sequence of outputs

    Ex. Image CaptioningGiven an image describe the image textually

  • RNN Paradigms

    Different problems are more suited for different RNN paradigms

    Given a single input predict a sequence of outputs

    Ex. Sentiment Analysis Given text predict positive or negative sentiment

  • RNN Paradigms

    Different problems are more suited for different RNN paradigms

    Given a sequence of inputs predict a sequence of outputs (of potentially different length)

    Ex. Machine TranslationGiven text in language A, translate it to language B

  • RNN Paradigms

    Different problems are more suited for different RNN paradigms

    Given a sequence of inputs predict a sequence of outputs (of the same length)

    Ex. Part of Speech TaggingGiven a sequence of words, label each word with its part of speech (Noun, Verb, etc)

  • How Layering RNNs WorkYou just stack them like this… Nothing special

    You can change the order or the direction

    You can change the granularity as well (Hierarchical Networks)

    The world is your oyster

  • Common Types of RNN Cells

    Different cells are better for different problems

    Dense Cell

    You aren’t limited to the above 3 layer structure, any feedforward style neural network architecture could work

    To calculate the output, simply perform the traditional feedforward network calculation

    Note, depending on the implementation “Output”

    and “Hidden1” may be the same thing

  • Common Types of RNN Cells

    Different cells are better for different problems

    Gated Recurrent Unit (GRU) Cell

    Where are the gates???

    The gates are the neural nets

    http://colah.github.io/posts/2015-08-Understanding-LSTMs/

  • Common Types of RNN Cells

    Different cells are better for different problems

    Gated Recurrent Unit (GRU) Cell Mathematics

    http://colah.github.io/posts/2015-08-Understanding-LSTMs/

  • Common Types of RNN Cells

    Different cells are better for different problems

    Long Short Term Memory (LSTM) Cell

    http://colah.github.io/posts/2015-08-Understanding-LSTMs/

    Cell Stateinput Cell Stateoutput

    Hidden Stateinput Hidden Stateoutput

  • Common Types of RNN Cells

    Different cells are better for different problems

    Long Short Term Memory (LSTM) Cell Mathematics

    http://colah.github.io/posts/2015-08-Understanding-LSTMs/

    Forget Gate

    Input Gate Output Gate

    Cell State Output

  • Common Pitfalls of RNNs

    ● These models can overfit incredibly easily. ○ Start with an incredibly simple model, with small gates and few

    layers, then expand.

    ● Vanishing/Exploding Gradients ○ Depending on your data, BTT can cause your gradients to

    become incredibly small (vanish) or become incredibly large (explode)■ Gated cells can mitigate this to an extent, but not entirely.■ Be sure to regularize and keep an eye on your gradients to

    see how they are doing

  • Conclusion

    Any Questions?


A Recurrent Quantum Neural Network

Deep Learning Recurrent Networksbhiksha/courses/deep... · Deep Learning Recurrent Networks 10/11/2017 1. Which open source project? Related math. What is it talking ... 7/03 8/03

Recurrent Neural Network tutorial (2nd)

Discovering Gated Recurrent Neural Network Architecturesnn.cs.utexas.edu/downloads/papers/AdityaRawalThesis.pdfDiscovering Gated Recurrent Neural Network Architectures by Aditya Rawal,

Hierarchical Recurrent Neural Networks for Long …papers.nips.cc/paper/1102-hierarchical-recurrent-neural...Hierarchical Recurrent Neural Networks for Long-term Dependencies 495 The

Neural Network Part 4: Recurrent Neural Networkspages.cs.wisc.edu/~jerryzhu/cs760/10_neural-networks-4.pdf · •recurrent neural networks (RNN) and the advantage •training recurrent

Online Learning with Stochastic Recurrent Neural …...recurrent neural network, hence, we refer to it as a stochastic recurrent neural network. The basis model consists of two different

RECURRENT NEURAL NETWORKS

Lecture 6: Recurrent & Graph Neural Networks · 2019. 12. 14. · UVA DEEP LEARNING COURSE –EFSTRATIOS GAVVES RECURRENT NEURAL NETWORKS - 1 Lecture 6: Recurrent & Graph Neural Networks

RECURRENT NEURAL NETWORKS - Lagout Intelligence/Neural networks... · Recurrent neural networks have been an interesting and important part of neural network research during the 1990's

Lecture 6: Recurrent Neural Networks - GitHub Pages · Lecture 6: Recurrent Neural Networks Efstratios Gavves. UVA DEEP LEARNING COURSE –EFSTRATIOS GAVVES RECURRENT NEURAL NETWORKS

Neural networks for structured data 1. Table of contents Recurrent models Partially recurrent neural networks Elman networks Jordan networks Recurrent

Recurrent Neural Networks - GitHub Pages

Dilated Recurrent Neural Networks - Neural …papers.nips.cc/paper/6613-dilated-recurrent-neural...Dilated Recurrent Neural Networks Shiyu Chang 1 , Yang Zhang , Wei Han 2 , Mo Yu

Recurrent neural networks for statistical machine translation (Neural Machine Translation) · 2016-11-14 · Recurrent neural networks for statistical machine translation (Neural

Neural and Evolutionary Computing - Lecture 5 1 Recurrent neural networks Architectures –Fully recurrent networks –Partially recurrent networks Dynamics

순환신경망(Recurrent neural networks) 개요

Tutorial Training Recurrent Neural Networks

Lecture 6: Recurrent & Graph Neural NetworksLecture 6: Recurrent & Graph Neural Networks Efstratios Gavves. UVA DEEP LEARNING COURSE –EFSTRATIOS GAVVES RECURRENT NEURAL NETWORKS

ELEC 677: Recurrent Neural Network Applications & Recurrent Neural Network … · 2016-11-08 · Recurrent Neural Network Applications & Recurrent Neural Network Language Models Lecture

Quasi-Recurrent Neural Networks - NVIDIAon-demand.gputechconf.com/...bradbury-quasi-recurrent-neural-netw… · Quasi-Recurrent Neural Networks James Bradbury and Stephen Merity Salesforce

Beyond Recurrent Neural Networks - GitHub Pagesiamthevastidledhitchhiker.github.io/figs/SOM_11MAR2016/TensorFlow... · The Hitchhiker’s Guide to TensorFlow: Beyond Recurrent Neural

Translation Modeling with Bidirectional Recurrent Neural ... · PDF fileTranslation Modeling with Bidirectional Recurrent Neural Networks ... g. language modeling, this type of neural

Deep Learning Recurrent Networksbhiksha/courses/deeplearning/... · Deep Learning Recurrent Networks 21/Feb/2018 1. Which open source project? 2. Related math. What is it talking

Recurrent neural network - SINTEF · 2017. 1. 18. · THE RECURRENT NEURAL NETWORK A recurrent neural network (RNN) is a universal approximator of dynamical systems. It can be trained

Jacek Mazurkiewicz, PhD Softcomputing · Softcomputing Part 3: Recurrent Artificial Neural Networks Self-Organising Artificial Neural Networks . Recurrent Artificial Neural Networks

Recurrent neural network - SINTEF · 2017-01-18 · THE RECURRENT NEURAL NETWORK A recurrent neural network (RNN) is a universal approximator of dynamical systems. It can be trained

Introduction to Deep Neural Networksbhiksha/courses/deep... · Introduction to Deep Neural Networks 0. Logistics Spring 2020 1. Neural Networks are taking over! •Neural networks

Deep Multi-State Dynamic Recurrent Neural Networks ...papers.nips.cc/paper/9594-deep-multi-state-dynamic-recurrent-neural... · Deep Multi-State Dynamic Recurrent Neural Networks

Recurrent neural networks and robust time series prediction - Neural

Recurrent Neural Network for Language Modeling Taskfraser/smt_nmt_2016_seminar/08_rnns.pdf · Recurrent Neural Network for Language Modeling Task ... Recurrent Neural Network Language

Particle Filter Recurrent Neural Networks

SUPERRESOLUTION RECURRENT CONVOLUTIONAL NEURAL NETWORKS

RECURRENT NEURAL NETWORKS - index-of.co.ukindex-of.co.uk/Artificial-Intelligence/Recurrent Neural Networks Design And... · recurrent neural networks and is exemplary of current research

Lecture 13: Recurrent Neural Nets