deploying enterprise deep learning masterclass preview - enterprise deep learning

Deploying Enterprise Deep Learning

Sam Putnam, Enterprise Deep Learning, LLC

July 27, 2017

Day 2

Sam Putnam

@edeeplearning

July 27


Deep Learning Made This Video

https://www.extremetech.com/g00/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they?i10c.referrer=https%3A%2F%2Fwww.google.com%2F

Preview - Slide Available at deeplearningconf.com

http://deeplearningconf.com

Sam Putnam

@edeeplearning


And This Song!



Sam Putnam

@edeeplearning

July 27


And Predicted These Housing Prices!

https://github.com/EnterpriseDeepLearning/housing-prices-wide-and-deep/blob/master/real_estate.ipynb

http://www.mirror.co.uk/news/uk-news/experts-predict-house-prices-could-10724816

Run the Code ->This Afternoon

Sam Putnam

@edeeplearning

July 27


OK, so what is Deep Learning?

http://p.migdal.pl/imgs/2017-04-30-learning-deep-learning/deep_learning_meme_keras.png



Sam Putnam

@edeeplearning

July 27


Deep Learning is Neural Networks

http://p.migdal.pl/imgs/2017-04-30-learning-deep-learning/deep_learning_meme_keras.png



Sam Putnam

@edeeplearning

July 27


Well, Artificial Neural Networks. But they are based off of how the brain works.

https://www.extremetech.com/g00/extreme/215170-artificial-neural-networks-are-changing-the-world-what-are-they?i10c.referrer=https%3A%2F%2Fwww.google.com%2F



Sam Putnam

@edeeplearning

July 27


This is a Neuron: Electrical Signals Are Sent By Axons and Received By Dendrites

http://www.newworldencyclopedia.org/entry/Dendrite



Sam Putnam

@edeeplearning

July 27


This is a Single Artificial Neuron With Two Inputs and Two Weights



Sam Putnam

@edeeplearning

July 27


The Artificial Neuron is Also Called a “Perceptron”. It “Perceives” Inputs.



Sam Putnam

@edeeplearning

July 27


b is the bias, and to activate the neuron:

Sam Putnam

@edeeplearning


Click Play, Visualize Neuron’s Performance

http://playground.tensorflow.org/

http://playground.tensorflow.org/

http://www.apple.com

Sam Putnam

@edeeplearning


Trust Me, When you add more inputs, your decision boundary becomes a “plane”



When you’re talking about Deep Learning you’re talking about Artificial Neural Networks

of Neurons

@edeeplearning



In particular, Deep Learning = Deep Artificial Neural Networks of Neurons

@edeeplearning



This it it - Just more than one layer of neurons between the input and output!

@edeeplearninghttp://blog.christianperone.com/2015/08/convolutional-neural-networks-and-feature-extraction-with-python/



You Tried The Line, Now Add A Hidden Layer To Capture More Complex Data Separations


This is No Line You Are Tweaking! This Is a More Complex Function




More Neurons + A Deeper Network = More Sophisticated Representations

@edeeplearninghttps://cloud.google.com/blog/big-data/2016/07/understanding-neural-networks-with-tensorflow-playground

Look at the last hidden layer - it is doing a pretty good job separating the data




This Architecture is called a Feedforward Neural Network




There’s a Lot Going on Here. Do You Really Want to Change Those Weights One at a Time?




Solution: Gradient Descent - Look How the Lowest Loss (0) Is Sought Out on the Right


What other Architectures Are There? Well, what about for Time Series Data

@edeeplearninghttps://deeplearning4j.org/lstm.html



Recurrent Neural Networks are Networks with Loops in Them

@edeeplearninghttp://colah.github.io/posts/2015-08-Understanding-LSTMs/



Information is passed from one step of the network to the next




Recurrent Neural Networks (RNNs) Contain A Single Layer (Yellow Box)


Recurrent Neural Networks (RNNs) cannot handle long term dependencies




LSTM Recurrent Neural Networks (RNNs) Contain Four Layers (Yellow Box)


Long Short Term Memory Networks (LSTMs) Can Handle Long Term Dependencies




@edeeplearning

Sigmoid Activation Function Scales Your Input Between 0 and 1

http://neuralnetworksanddeeplearning.com/chap4.html



Sigmoid Activation Function Scales Your Input Between 0 and 1

@edeeplearning





Tanh Activation Function Scales Your Input Between -1 and 1

@edeeplearning





RELU Activation Function Cuts off Negative Inputs

@edeeplearning





What About Image Data? For That, Convolutional Neural Networks (Right)

@edeeplearninghttp://cs231n.github.io/convolutional-networks/

Convolution Just Means Picking Out Representative Elements from a Bunch of

Pixels (F represents all the below)

@edeeplearning



This is What Convolution Looks Like, a Little Boring, But Look How It Reduces the Size of

the Data (3x3 on the Right!)

@edeeplearning



So Convolution Is a Mathematical Operation (Using a Weight Filter, Let’s Look!), but Even Simpler is

Max Pooling, just take the Biggest Pixel!

@edeeplearning

A Pre-Trained Image Recognition Neural Network Looks Like This (Input At Left) - Let’s

Zoom In

@edeeplearninghttps://arxiv.org/pdf/1409.4842.pdf



Sam Putnam

@edeeplearning

July 27


Deep Learning in Production Considerations

Help me find the source for this nice diagram? Lost it :(

Sam Putnam

@edeeplearning

July 27


Solution - Use What I Know and Am Allowed To Use, i.e. Kaggle, AWS, TF, Jupyter

Help me find the source for this nice diagram? Lost it :(



Sam Putnam

@edeeplearning

July 27


What I Did - A Real Estate Agent Told Me About His Problem & Pointed Me To The Data

https://www.fortunebuilders.com/how-to-become-a-real-estate-agent/



Sam Putnam

@edeeplearning

July 27


I Took a Holdout Set of the Data and Left the Rest for Train/Validate

https://www.fortunebuilders.com/how-to-become-a-real-estate-agent/

Sam Putnam

@edeeplearning

July 27


Started with Feature Selection and then Linear Regression in Good old Excel, WITH the subject matter

expert (the real estate agent). Did Pretty Well.

https://www.youtube.com/watch?v=SQkpLMLoqww



Sam Putnam

@edeeplearning

July 27



How Well? 18 percent Mean Error and 26 percent Median Error before Feature Selection using Weighted Importance, Did some thinking:


Sam Putnam

@edeeplearning

July 27


Feature Importance, for Feature Selection, Often Use After Deep Learning Step, to Gain Interpretability

Sam Putnam

@edeeplearning

July 27


https://www.kaggle.com/samdeeplearning/naive-subsample-5-10-city/output

GBM (Boosted Decision Tree) - with Hyperparameter Tweaking - Yields Strong Model for this Problem

Sam Putnam

@edeeplearning

July 27


https://www.kaggle.com/samdeeplearning/naive-subsample-0-25-xgb/notebook


Starts to Degrade Upon Subsampling only 1/4 of the Trees


Sam Putnam

@edeeplearning

July 27


Sam Putnam

@edeeplearning

July 27


Want To See If Neural Network Can Improve Performance

Sam Putnam

@edeeplearning

July 27


Building off of a preexisting housing prices regression model




Sam Putnam

@edeeplearning

July 27


Set Input Layer (n_hidden_1 is input layer here) to number of features (27)




Sam Putnam

@edeeplearning

July 27


Combining the features right off the bat by using a wide first hidden layer (200 nodes)




Sam Putnam

@edeeplearning

July 27


Set number of classes to 1 for regression problem




Sam Putnam

@edeeplearning

July 27


Multilayer Perceptron = Feedforward Perceptron We Looked At Before




Sam Putnam

@edeeplearning

July 27


RELU - As Mentioned in Executing Strategies Yesterday, not using negative features/weights


Sam Putnam

@edeeplearning

July 27


Training - Can tell performance needs tweaking, pretty good on houses 2-6


123456

Estimate on LeftPrediction on Left

Sam Putnam

@edeeplearning

July 27


Let’s Dive Into the TensorFlow Housing Code

https://www.kaggle.com/samdeeplearning/deep-neural-network-for-starters-r/edit



Sam Putnam

@edeeplearning

July 27


What Next? Go Deep Learning!

https://www.youtube.com/watch?v=cSKfRcEDGUs&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal&index=6

Sam Putnam

@edeeplearning

July 27


1) Identify an architecture for your problem


Sam Putnam

@edeeplearning

July 27


2) Identify a Framework, Be Willing to Try a New Framework

https://blog.algorithmia.com/deploying-deep-learning-cloud-services/

Sam Putnam

@edeeplearning

July 27


3) Build or tailor a model for your application, Don’t be Afraid to Reference Academic Code


Sam Putnam

@edeeplearning

July 27


4)Tweak the Parameters for your Model


Sam Putnam

@edeeplearning

July 27


5)Train on a (probably, Nvidia, maybe Intel) GPU if need be (for image data, you need it)

https://www.youtube.com/watch?v=2NrgPdGSXhE

https://www.youtube.com/watch?v=2NrgPdGSXhE

Sam Putnam

@edeeplearning

July 27


6) Validate, test, check in the real world. Iterate!

https://medium.com/towards-data-science/train-test-split-and-cross-validation-in-python-80b61beca4b6



Sam Putnam

@edeeplearning

July 27


Weekend trial: Josh Gordon’s Machine Learning Recipes. If the code doesn’t work for you (I am 99 %

sure it will), let me know!




Sam Putnam

@edeeplearning

July 27


Like R? Try this notebook on my Kaggle


Sam Putnam

@edeeplearning

July 27


Submit your predictions, even! (disclaimer: middle of the pack result)




Sam Putnam

Thank you to Google and others who have published diagrams and photos. Slides are for today only.

@edeeplearning

Questions/Comments: [email protected]

Thank you

July 27


deploying enterprise deep learning masterclass preview - enterprise deep learning

Software