modern frameworks for machine learning
TRANSCRIPT
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Machine Learning in practice
Tools and frameworks
Sergii [email protected]
Samsung R&D Institute Ukraine
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Machine Learning Workflow We need to go deeper
First generation of NN frameworks Modern NN framework
Speculations and summary
Content
2A sample title
A sample title 3
Machine LearningWorkflow
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Good beginning
4Machine Learning Workflow
Good Architecture
Machine Learning Workflow 5
– Experiments– Datasets– Trained models– Notebooks
– REST API– Other
cognitiveservices
A bit more advanced options
Machine Learning Workflow 6
Wolfram Data Science platform (Expected soon)
AmazoneMachine Learning
Sergii Nechuiviter. Samsung R&D Institute Ukraine Machine Learning Workflow 7
ML Framework Implementation Pros
scikit-learn Python The most popular, standard de-facto
Accord .Net Framework .Net C# Best for C#
mlpack C++ Fastest on CPU
MLlib from Spark Scala Distributed out of the box
RapidMiner Java Vast pack of algorithmsGUI workflow designer
Generic machine learning tools
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Too much data to analyze?
8A sample title
Sergii Nechuiviter. Samsung R&D Institute Ukraine
First generation ofNN frameworks
A sample title 9
Sergii Nechuiviter. Samsung R&D Institute Ukraine
CaffeBerkeley Vision and Learning Center
Specialized on convolutional NN
Configs, Python
No cluster support
GPU support
10A sample title
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Torch7Facebook
Imperative
LuaJIT + Python
No cluster support
GPU support
Embeddable, with ports to iOS, Android and FPGA backends
11A sample title
Sergii Nechuiviter. Samsung R&D Institute Ukraine
TheanoInitiated by LISA lab, now a vast community
Computational graph optimizations: generic and for RNNPython
No cluster support
GPU support
Pros: Easy NN with Keras, Lasagne and Blocks; RNN optimizations
Cons: Low-level, Error messages can be unhelpful
12A sample title
Sergii Nechuiviter. Samsung R&D Institute Ukraine
ModernNN frameworks
Modern NN frameworks 13
DL4J
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Deeplearning4JSkymind
NN specific ? Fast tensor backend
Java, Scala, Clojure
Spark, Hadoop, Mesos, Akka
Multi-GPU
Embeddable due to Java
Pros: Enterprise support
DL4J
14Modern NN frameworks
Sergii Nechuiviter. Samsung R&D Institute Ukraine
TheanoInitiated by LISA lab, now a vast community
Computational graph optimizations: generic and for RNNPython
No cluster support
GPU support
Pros: Easy NN with Keras, Lasagne and Blocks; RNN optimizations
Cons: Low-level, Error messages can be unhelpful
15Modern NN frameworks
Sergii Nechuiviter. Samsung R&D Institute Ukraine
What is computational graph?
Modern NN frameworks 16
• Automatic differentiation
• Automatic parallelization
• Merging of similar subgraphs
• Arithmetic simplification ()
• Improvements to numerical stability
Modern NN frameworks 17
Computational graph optimizations
Sergii Nechuiviter. Samsung R&D Institute Ukraine
TensorFlowGoogle
Computational graph optimizations: generic and for RNNC++, Python
From V0.7 cluster support
From V0.7 multi-GPU
Linux and ARM deployment; No Windows
Google Cloud Machine Learning, Rescale
Pros: TensorBoard for visualization
Cons: slow and bulk
18Modern NN frameworks
Sergii Nechuiviter. Samsung R&D Institute Ukraine
CNTKMicrosoft
Computational graph optimizations: generic and for RNNNetwork Description Language + C/C++
Cluster support
Multi-GPU
?
Azure Machine Learning
Pro: fastest at ImageNet LSVR 2015
19Modern NN frameworks
Sergii Nechuiviter. Samsung R&D Institute Ukraine
ImageNet LSVR 2015
20Modern NN frameworks
Sergii Nechuiviter. Samsung R&D Institute Ukraine
NeonNervana™
?
Python
Cluster
Multi-GPU
?
Nervana™ Cloud
Pros: fastest convolutions
21Modern NN frameworks
Sergii Nechuiviter. Samsung R&D Institute Ukraine
convnet-benchmarks
GoogleNet V1 - Input 128x3x224x2246-core Intel Core i7-5930K CPU @ 3.50GHz + NVIDIA Titan X + Ubuntu 14.04 x86_64https://github.com/soumith/convnet-benchmarks
Neon TensorFlow Torch Chainer Caffe0
500
1000
1500
2000
72 135 112 189
786
157310 349
497
1148
forward (ms) backward (ms)
Exec
ution
tim
e, m
s
22Modern NN frameworks
Sergii Nechuiviter. Samsung R&D Institute Ukraine
MXNetDMLC
Computational graph optimizations: generic and for RNN + ImperativePython/R/Julia/Go/C++
Cluster support
Multi-GPU support
Windows, Linux, Mobile, Apple
AWS recommended
Pros: versatile, fast as Torch and Theano
23Modern NN frameworks
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Theano DL4J TensorFlow CNTK Neon MXNet
Creator – Skymind Google Microsoft Nervana™ DMLCComputational Graph V ? V V ? V
Cluster support – V V V V V
GPUSupport 1 Multi Multi Multi Multi Multi
Mobile – may be V ? ? VCloud – V V V V VPerformance Ok ? Slow Fast Fast Ok
Modern NN frameworks
A sample title 24
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Speculations and summary
Speculations and summary 25
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Analogcomputers
Specialized digital computers
Generic computers
Generic computers with coprocessors
Generic computer with coprocessor
GPU computations
Why deep learning happened?
Rise of single CPU performance Distributed
computing
Single core reach plateau
of performancePrice of computations
is dropping
26Speculations and summary
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Analogcomputers
Specialized digital computers
Generic computers
Generic computers with coPU
Generic computer with coPU
GPU computa-tions
2016:NN on chip
Quantum PU?
Why deep learning happened?
Rise of single CPU performance Distributed
computing
Single core reach plateau
of performancePrice of computations is dropping
?
?
27Speculations and summary
Human-Machine cooperation
?
Sergii Nechuiviter. Samsung R&D Institute Ukraine
Try Microsoft Azure Machine Learning FreeTrialTry scikit-learnKnow about computational graphsTry MXNet
Be ready for changes!
Summary
Speculations and summary 28
A sample title 29