deep learning in action - recurrent null · why features matter 4 2017/15/09 deep learning in...
TRANSCRIPT
![Page 1: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/1.jpg)
BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
Deep Learning in ActionWith DL4J!
Sigrid Keydana
![Page 2: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/2.jpg)
Deep Learning in action - with DL4J2 2017/15/09
Deep Learning in 3 W’s
![Page 3: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/3.jpg)
What?
Deep Learning in action - with DL4J3 2017/15/09
Pre-ML programs
in: data
in: rules
out: conclusion
Machine Learning (ML)
in: data
learn: mappings/functions
out: conclusion
Deep Learning (DL)
in: data
learn: features
learn: mappings/functions
out: conclusion
![Page 4: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/4.jpg)
Why features matter
Deep Learning in action - with DL4J4 2017/15/09
Source: Goodfellow et al., Deep Learning, 2016
![Page 5: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/5.jpg)
Example: Features for object classification
Deep Learning in action - with DL4J5 2017/15/09
Source: Goodfellow et al., Deep Learning, 2016
![Page 6: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/6.jpg)
hoW?
Deep Learning in action - with DL4J6 2017/15/09
Learning from errors in a deep network
Source: Wikipedia Source: Goodfellow et al., Deep Learning, 2016
![Page 7: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/7.jpg)
Why?
Deep Learning in action - with DL4J7 2017/15/09
$$$
• Google• Microsoft• Facebook• Baidu• Apple• Nvidia• Intel• IBM • . . .
Applications in industry
• Autonomous driving• Machine translation• Image recognition• Manufacturing• Robotics• Healthcare• Music, text, image generation• Chatbots• . . .
Applications in science
• Finance• Physics• Cosmology• Pharmacology• Medicine• Meteorology• Biology• Engineering• . . .
![Page 8: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/8.jpg)
Deep Learning in action - with DL4J8 2017/15/09
So is this just for the big guys?
![Page 9: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/9.jpg)
No.
Deep Learning in action - with DL4J9 2017/15/09
Just need
Reasonable amount of data Adequate hardware – or run in the cloud! Deep learning software – and people who know to code and run it
Let’s zoom in on the software.
![Page 10: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/10.jpg)
Deep Learning frameworks
Deep Learning in action - with DL4J10 2017/15/09
Python (possibly C/C+
+)
Keras
PyTorch
TheanoTensorFlow
MXNet
![Page 11: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/11.jpg)
What if my whole environment is Java/JVM-based?
Deep Learning in action - with DL4J11 2017/15/09
And/or I need to run on a (Spark/Hadoop) cluster?
And/or I need to optimize performance?
Enter…
![Page 12: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/12.jpg)
Deep Learning in action - with DL4J12 2017/15/09
DL4J
![Page 13: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/13.jpg)
DL4J in a nutshell
Deep Learning in action - with DL4J13 2017/15/09
• Open-source DL framework for the JVM ecosystem (maintained by Skymind)
• Distributed – runs on Spark and Hadoop
• Fast (with adequate JVM settings)
• Lots of nicely commented example code
• Helpful developers! (Github / Gitter chat)
![Page 14: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/14.jpg)
So if the focus is on production …
Deep Learning in action - with DL4J14 2017/15/09
… what does the developer’s experience look like?
• Developer 1 (normally uses Keras for quick experiments): “I miss playing around in the notebook… What’s actually going on in that code?”
• Developer 2 (too impatient for fiddling around with parameters): “Is that endless training time really necessary? Can’t I just use something out-of-the-box?”
• Developer 3 (could be me): “I want to apply this method from that paper and don’t have the time… perhaps it’s already implemented in …?”
![Page 15: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/15.jpg)
But first… let’s just see some code!
Deep Learning in action - with DL4J15 2017/15/09
How does linear regression look like in DL4J?
MultiLayerNetwork net = new MultiLayerNetwork(new NeuralNetConfiguration.Builder() .iterations(numEpochs) .optimizationAlgo(OptimizationAlgorithm.CONJUGATE_GRADIENT) .updater(Updater.SGD) .list() .layer(0, new DenseLayer.Builder().nIn(numFeatures).nOut(hiddenDim) .activation(Activation.RELU) .build()) .layer(1, new OutputLayer.Builder(LossFunctions.LossFunction.MSE) .activation(Activation.IDENTITY) .nIn(hiddenDim).nOut(outputDim).build()) .build());net.init();net.fit(X, Y);
![Page 16: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/16.jpg)
Deep Learning in action - with DL4J16 2017/15/09
Spot on … agile development
![Page 17: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/17.jpg)
Keras model import
Deep Learning in action - with DL4J17 2017/15/09
• Can experiment in Keras and then import models
• Let’s see this in action!
MultiLayerNetwork network = KerasModelImport.importKerasSequentialModelAndWeights(modelPath);
![Page 18: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/18.jpg)
Keras model import demo: Crack? No crack?
Deep Learning in action - with DL4J18 2017/15/09
![Page 19: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/19.jpg)
Deep Learning in action - with DL4J19 2017/15/09
Spot on … pretrained models
![Page 20: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/20.jpg)
Model zoo
Deep Learning in action - with DL4J20 2017/15/09
• Can use e.g. VGG16, VGG19, ResNet, GoogleNet… with weights mostly trained on ImageNet
• Examples available for different forms of usage (adaptation to own classes, fine tuning)
• Let’s try this out!
ZooModel vgg16 = new VGG16();ComputationGraph pretrainedNet = (ComputationGraph) vgg16.initPretrained(PretrainedType.IMAGENET);
![Page 21: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/21.jpg)
VGG 16 demo: What’s this?
Deep Learning in action - with DL4J21 2017/15/09
Source: https://www.juggle.org/claude-shannon-mathematician-engineer-genius-juggler/
Source: http://krisholm.com/en/blog/Introducing-the-KH-27.5
![Page 22: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/22.jpg)
Deep Learning in action - with DL4J22 2017/15/09
Spot on … available algorithms
![Page 23: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/23.jpg)
Example: Anomaly detection
Deep Learning in action - with DL4J23 2017/15/09
• One state-of-the-art approach: Variational Autoencoder with reconstruction probability
• Implemented in DL4J as a specific variational layer
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder() .//… .layer(0, new VariationalAutoencoder.Builder() .activation(Activation.LEAKYRELU) .encoderLayerSizes(encoderSizes) .decoderLayerSizes(decoderSizes) .pzxActivationFunction(latentActivation) .reconstructionDistribution(new BernoulliReconstructionDistribution(Activation.SIGMOID)) .nIn(inputSize) .nOut(latentSize) .build()) .pretrain(true).backprop(false).build();
![Page 24: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/24.jpg)
Variational autoencoder demos
Deep Learning in action - with DL4J24 2017/15/09
UNSW-NB15 intrusion detectionMNIST handwritten digits
![Page 25: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/25.jpg)
Conclusion: DL4J
Deep Learning in action - with DL4J25 2017/15/09
• Nice, nicely documented, actively developed
• Many architectures and pretrained models available
• Instructive example code
![Page 26: Deep Learning in Action - recurrent null · Why features matter 4 2017/15/09 Deep Learning in action - with DL4J Source: Goodfellow et al., Deep Learning, 2016](https://reader031.vdocuments.net/reader031/viewer/2022022015/5b5ac0cc7f8b9a302a8c7a1e/html5/thumbnails/26.jpg)
Questions? Thank you!
2017/15/09 Deep Learning in action - with DL4J26