Upload File

sequences + nlp deep learning series (wwc …...talk outline •deep learning intro •distilled...

  • Home
  • Documents
  • SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What
29
DEEP LEARNING SERIES (WWC-AI) Tim Scarfe Machine learning appreciator from the UK! http://aka.ms/mdml MACHINE LEARNING @ MICROSOFT

Upload: others

Post on 17-Jun-2020

11 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

DEEP LEARNING SERIES (WWC-AI)

Tim ScarfeMachine learning appreciator from the UK!

http://aka.ms/mdml

MACHINE LEARNING @ MICROSOFT

Page 2: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

TALK OUTLINE• Deep learning intro

• Distilled concepts of deep learning

• Why are neural networks good at sequence processing?

• What is sequence processing?

• Working with text data

• Recurrent neural networks

• 1d convolutional neural networks

Page 3: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

WHAT IS A NEURAL NETWORK?

Page 4: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

WHAT IS A DEEP NEURAL NETWORK?

Page 5: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

DISTILLED CONCEPTS OF DEEP LEARNING

Page 6: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

ENTIRE MACHINE IS TRAINABLE

• The networks have many levels of depth

• Machine learns a hierarchy of representations

• No feature extraction required

Traditional ML

Feature Extractor Trainable Classifier

Deep Learning

Low Level Features Trainable ClassifierMid Level Features High Level Features

Representations are hierarchical and trained automatically

Hand crafted features

Page 7: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

UNIVERSAL FUNCTION APPROXIMATORS

• Unlike other shallow ML algorithms; you can map between data domains

SEQUENCES

We learn a spatial transformation between the twoSPEECH

LANGUAGE

IMAGES

[…]

SEQUENCES

SPEECH

LANGUAGE

IMAGES

[…]

Page 8: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

NATIVE DATA-DOMAIN FEATURES

Unlike other algorithms, NNs can natively encode useful and obvious relationships in the data domain

• Local spatial dependencies (vision)

• Time dependencies (language, speech)

Recurrent Neural Networks

Convolutional Neural Networks

Page 9: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

COMPOSABILITY

• Composability

• Deep Learning research is very applied

• Accessibility

• Software analogy

Page 10: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

WHAT IS A SEQUENCE?

Page 11: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

WHAT IS SEQUENCE PROCESSING• RNNs

• Timeseries classification

• Anomaly detection in timeseries

• Entity recognition

• Revenue forecasting

• Question + Answer

• 1d Convnets

• Spelling correction

• Document classification

• Machine translation

Page 12: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

WHAT IS SEQUENCE PROCESSING• RNNs

• When global order matters

• 1d Convnets

• Speed

• Local temporal dependencies

• You can stack them!

Page 13: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

TOKENIZATION

• Words

• Characters

• N-grams

N-Grams example

Page 14: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

WORD VECTORS VS WORD EMBEDDINGS?

Page 15: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

WORD EMBEDDINGS

• Word2Vec

• GloVe

Page 16: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

RECURRENT NEURAL NETWORKS

Page 17: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

RECURRENT NEURAL NETWORKS

Image © Francois Chollet

Page 18: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

LSTMS(1)

Image © Francois Chollet

Page 19: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

LSTMS(2)

Page 20: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

LSTMS(3)

Image © Francois Chollet

Page 21: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

LSTM VS GRU

LSTM

GRU

Page 22: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

BI-DIRECTIONAL LSTMS

Page 23: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

1D-CNNS

Page 24: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

2DCNNS – SAME CONCEPT

Low Level Features Mid Level Features

High Level Features

Page 25: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

STACKING IS COOL

Page 26: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

UNIVERSAL MACHINE LEARNING PROCESS

• Define problem

• Define success

• Validation process

• Vectorise/normalize data

• Develop naïve model

• Refine model based on validation performance

Page 27: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

DEMOS OF RNNS

Page 28: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

https://www.manning.com/books/deep-learning-with-python

Page 29: SEQUENCES + NLP DEEP LEARNING SERIES (WWC …...TALK OUTLINE •Deep learning intro •Distilled concepts of deep learning •Why are neural networks good at sequence processing? •What

THANK YOU!


THE DEEP LEARNING REVOLUTION - Nvidia...THE DEEP LEARNING REVOLUTION Superhuman Breakthroughs in Modern Artificial Intelligence Powered by GPUs JOIN THE DEEP LEARNING ERA DEEP LEARNING

Tutorial: Deep Reinforcement Learning · Outline Introduction to Deep Learning Introduction to Reinforcement Learning Value-Based Deep RL Policy-Based Deep RL Model-Based Deep RL

Deep Learning for Laser Based Odometry Estimation - …juxi.net/workshop/deep-learning-rss-2016/papers/Nicolai - Deep... · Deep Learning for Laser Based Odometry Estimation ... deep

Einführung ins Deep Learning · Einführung ins Deep Learning deep space computing AG 6. Dezember 2019 deep-space.ch

Introducing Deep Learning with MATLAB - Systematics · Introducing Deep Learning with MATLAB6 Inside a Deep Neural Network ... Deep learning is a subtype of machine learning. With

Learning Deep Learning

Wwc templates

Lecture 1: Introduction to Deep Learning · uva deep learning course –efstratios gavves introduction to deep learning - 1

Introducing Deep Learning with MATLAB - MathWorks · Introducing Deep Learning with MATLAB7 How A Deep Neural Network ... relevant features of an image. With deep learning, ... Deep

Deep Learning - Oxford Statisticspalamara/teaching/SML19/HT19_lecture15.pdf · SB2b/SM4 - Deep Learning Deep Learning Infrastructure! Computational Infrastructure critical to deep

Demystifying Deep Learning - MathWorks · Demystifying Deep Learning. 2 What’s Deep Learning and why should I care? A practical approach to Deep Learning (for images) –Transfer

Binary Deep Learning - TAUweb.eng.tau.ac.il/.../wp-content/uploads/2017/03/Binary-Deep-Learning.pdf · Binary Deep Learning Deep Learning Seminar, School of Electrical Engineering,

CS 678 – Deep Learning1 Deep Learning Early Work Why Deep Learning Stacked Auto Encoders Deep Belief Networks

Lecture 1: Introduction to Deep Learning - uvadlc.github.io · Lecture 1: Introduction to Deep Learning Efstratios Gavves. UVA DEEP LEARNING COURSE UVA DEEP LEARNING COURSE –EFSTRATIOS

Angular wwc

lapres wwc

Deep learning / representation learning

Deep Learning for Reinforcement Learning in Pacman · Deep Learning for Reinforcement Learning in Pacman Deep Learning für Reinforcement Learning in Pacman Vorgelegte Bachelor-Thesis

Deep Learning I - Korea Universitydsba.korea.ac.kr/wp/wp-content/seminar/Deep learning/Deep Learning... · Deep Learning I Deep learning is research on learning models with multilayer

Tutorial: Deep Reinforcement Learning - Machine Learning ...hunch.net/~beygel/deep_rl_tutorial.pdfTutorial: Deep Reinforcement Learning - Machine Learning

Hangzhou Deep Learning Meetup-Deep Reinforcement Learning

Introduction to Deep Learning - Joyce Hojoyceho.github.io/cs534_s17/slide/25-deep-learning.pdf · Deep Learning: Overview ... Review: Backpropogation https: ... • Deep Learning

Intro to Deep Learning - bpTT Bright Minds … · 17/08/2020  · Intro to Deep Learning Deep Sequence Modeling Deep Computer Vision Deep Generative Modeling Deep Learning Applications

Deep Learning Early Work Why Deep Learning Stacked Auto Encoders Deep Belief Networks CS 678 – Deep Learning1

From Reinforcement Learning to Deep Reinforcement …fagostin/assets/files/...Keywords: Machine learning · Reinforcement learning Deep learning · Deep reinforcement learning 1 Introduction

Deep Learning - South Dakota State UniversityDeep Learning Deep learning (also known as deep structured learning, hierarchical learning or deep machine learning) is a branch of machine

UNSUPERVISED DEEP LEARNING - TAUweb.eng.tau.ac.il/.../uploads/2016/12/Unsupervised-Deep-Learning.pdf · UNSUPERVISED DEEP LEARNING Erez Aharonov Noam Eilon Deep Learning Seminar School

WWC Intervention Report U.S. DEPARTMENT OF EDUCATION … · that falls within the scope of the Students with Learning Disabilities review protocol meets What Works Clearinghouse (WWC)

Tokheimmid.tokheim.com/pdf/TOKHEIM-EVAL-C012-WWC-TC7138-REV56V.pdf · 2016. 1. 6. · WWC-EIN 3.21 (checksum: OxD7FD) WWC-IFSF 3.34 (checksum Ox8CA9) WWC-IFSF 3.35 (checksum Ox6FD6)

NPDL Global Report - New Pedagogies for Deep Learning · Deep Learning 7 The 6Cs 7 Deep Learning Progressions 9 Impact 16 Key Findings: Deep Learning 18 New Pedagogies 19 Learning

Laporan WWC

Deep Learning in real world @Deep Learning Tokyo

Deep materials informatics: Applications of deep learning in … · deep learning: † Deep learning requires big data: In many cases, the biggest limiting factor for applying deep

Lecture 3: Deep Learning Optimizations - GitHub Pagesuvadlc.github.io/lectures/nov2019/lecture3.pdf · Lecture 3: Deep Learning Optimizations Deep Learning @ UvA. UVA DEEP LEARNING

Deep Learning for Generic Object Detection: A Survey · deep learning, probabilistic models, autoencoders, manifold learning, and deep networks 17 Deep learning LeCun et al. (2015)