squeezing deep learning into mobile phones

Download Squeezing Deep Learning Into Mobile Phones

Post on 21-Mar-2017




1 download

Embed Size (px)


PowerPoint Presentation

Squeezing Deep Learning into mobile phones - A Practitioners guideAnirudh Koul



Anirudh Koul , @anirudhkoul , http://koul.aiProject Lead, Seeing AIApplied Researcher, Microsoft AI & ResearchAkoul at Microsoft dot com

Currently working on applying artificial intelligence for productivity, augmented reality and accessibility

Along with Eugene Seleznev, Saqib Shaikh, Meher Kasam


Why Deep Learning On Mobile?iLatency



Mobile Deep Learning RecipeiMobile Inference Engine + Pretrained Model = DL App




Building a DL App in _ time


Building a DL App in 1 hour

No, dont do it right now. Do it in the next session.6

Use Cloud APIsiMicrosoft Cognitive ServicesClarifaiGoogle Cloud VisionIBM Watson ServicesAmazon Rekognition


Microsoft Cognitive ServicesiModels won the 2015 ImageNet Large Scale Visual Recognition ChallengeVision, Face, Emotion, Video and 21 other topics


Building a DL App in 1 day




Energy to trainConvolutionalNeural NetworkEnergy to useConvolutionalNeural Network


Base PreTrained ModeliImageNet 1000 Object CategorizerInceptionResnet


Running pre-trained models on mobileiMXNet TensorflowCNNDroidDeepLearningKitCaffeTorch

Speedups : No need to decode JPEGs, directly deal with camera image buffers12

MXNETiAmalgamation : Pack all the code in a single source file

Pro:Cross Platform (iOS, Android), Easy portingUsable in any programming language

Con:CPU only, Slow


Very memory efficient. MXNet can consume as little as 4 GB of memory when serving deep networks with as many as 1000 layers

Deep learning (DL) systems are complex and often have a few of dependencies. It is often painful to port a DL library into different platforms, especially for smart devices. There is one fun way to solve this problem: provide a light interface and putting all required codes intoa single filewith minimal dependencies.The idea of amalgamation comes from SQLite and other projects, which pack all code into a single source file. To create the library, you only need to compile that single file. This simplifies porting to various platforms. Thanks toJack Deng, MXNet provides anamalgamationscript, that compiles all code needed for prediction based on trained DL models into a single.ccfile, which has approximately 30K lines of code. The only dependency is a BLAS library.The compiled library can be used by any other programming language.By using amalgamation, we can easily port the prediction library to mobile devices, with nearly no dependency. Compiling on a smart platform is no longer a painful task. After compiling the library for smart platforms, the last thing is to call C-API in the target language (Java/Swift).This does not use GPU.It mentions about dependency on BLAS, because of which it seems it uses CPU on mobile

BLAS (Basic Linear Algebraic Subprograms) is at the heart of AI computation. Because of the sheer amount of number-crunching involved in these complex models the math routines must be optimized as much as possible. The computational firepower of GPUs make them ideal processors for AI models.

It appears that MXNet can use Atlas (libblas), OpenBLAS, and MKL. These are CPU-based libraries.

Currently the main option for running BLAS on a GPU is CuBLAS, developed specifically for NVIDIA (CUDA) GPUs. Apparently MXNet can use CuBLAS in addition to the CPU libraries.

The GPU in many mobile devices is a lower-power chip that works with ARM architectures which doesn't have a dedicated BLAS library yet.

what are my other options?

Just go with the CPU. Since it's the training that's extremely compute-intensive, using the CPU for inference isn't the show-stopper you think it is. In OpenBLAS, the routines are written in assembly and hand-optimized for each CPU it can run on. This includes ARM.

Using a C++-based framework like MXNet is probably the best choice if you are trying to go cross-platform.


TensorflowiEasy pipeline to bring Tensorflow models to mobileGreat documentationOptimizations to bring model to mobileUpcoming : XLA (Accelerated Linear Algebra) compiler to optimize for hardware


CNNdroidiGPU accelerated CNNs for AndroidSupports Caffe, Torch and Theano models~30-40x Speedup using mobile GPU vs CPU (AlexNet)

Internally, CNNdroid expresses data parallelism for different layers, instead of leaving to the GPUs hardware scheduler

Different methods are employed in acceleration of different layers in CNNdroid. Convolution and fully connected layers, which are data-parallel and normally more compute intensive, are accelerated on the mobile GPU using RenderScript framework.

A considerable portion of these two layers can be expressed as dot products. The dot products are more efficiently calculated on SIMD units of the target mobile GPU. Therefore, we divide the computation in many vector operations and use the pre-defined dot function of the RenderScript framework. In other words, we explicitly express this level of parallelism in software, and as opposed to CUDA-based desktop libraries, do not leave it to GPUs hardware scheduler. Comparing with convolution and fully connected layers, other layers are relatively less compute intensive and not efficient on mobile GPU. Therefore, they are accelerated on multi-core mobile CPU via multi-threading. Since ReLU layer usually appears after a convolution or fully connected layer, it is embedded into its previous layer in order to increase the performance in cases where multiple images are fed to the CNNdroid engine15

DeepLearningKitiPlatform : iOS, OS X and tvOS (Apple TV)DNN Type : CNNs models trained in CaffeRuns on mobile GPU, uses MetalPro : Fast, directly ingests Caffe modelsCon : Unmaintained


CaffeiCaffe for Androidhttps://github.com/sh1r0/caffe-android-libSample apphttps://github.com/sh1r0/caffe-android-demo

Caffe for iOS : https://github.com/aleph7/caffeSample apphttps://github.com/noradaiko/caffe-ios-samplePro : Usually couple of lines to port a pretrained model to mobile CPUCon : Unmaintained

Mostly community contributions, not part of the main app17

Running pre-trained models on mobileiMobile LibraryPlatformGPUDNN Architecture SupportedTrained Models SupportedTensorflowiOS/AndroidYesCNN,RNN,LSTM, etcTensorflowCNNDroidAndroidYesCNNCaffe, Torch, TheanoDeepLearningKitiOSYesCNNCaffeMXNetiOS/AndroidNoCNN,RNN,LSTM, etcMXNetCaffeiOS/AndroidNoCNNCaffeTorchiOS/AndroidNoCNN,RNN,LSTM, etcTorch


Building a DL App in 1 week



Learn Playing an Accordion3 months



Learn Playing an Accordion3 monthsKnows PianoFine Tune Skills1 week


I got a dataset, Now What?iStep 1 : Find a pre-trained modelStep 2 : Fine tune a pre-trained modelStep 3 : Run using existing frameworksDont Be A Hero - Andrej Karpathy


How to find pretrained models for my task?iSearch Model Zoo

Microsoft Cognitive Toolkit (previously called CNTK) 50 ModelsCaffe Model ZooKerasTensorflowMXNet


AlexNet, 2012 (simplified)i

[Krizhevsky, Sutskever,Hinton12]

Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Ng, Unsupervised Learning of Hierarchical Representations with Convolutional Deep Belief Networks, 11n-dimensionFeaturerepresentation

Learned hierarchical features from a deep learning algorithm. Each feature can be thought of as a filter, which filters the input image for that feature (a nose). If the feature is found, the responsible units generate large activations, which can be picked up by the later classifier stages as a good indicator that the class is present.24

Deciding how to fine tuneiSize of New DatasetSimilarity to Original DatasetWhat to do?LargeHighFine tune.SmallHighDont Fine Tune, it will overfit.Train linear classifier on CNN FeaturesSmallLowTrain a classifier from activations in lower layers.Higher layers are dataset specific to older dataset.LargeLowTrain CNN from scratch


In practice, we dont usually train an entire DCNN from scratch with random initialization. This is because it is relatively rare to have a dataset of sufficient size that is required for the depth of network required. Instead, it is common to pre-train a DCNN on a very large dataset and then use the trained DCNN weights either as an initialization or a fixed feature extractor for the task of interest.

Fine-Tuning: Transfer learning strategies depend on various factors, but the two most important ones are the size of the new dataset, and its similarity to the original dataset. Keeping in mind that DCNN features are more generic in early layers and more dataset-specific in later layers, there are four major scenarios:New dataset is smaller in size and similar in content compared to original dataset: If the data is small, it is not a good idea to fine-tune the DCNN due to overfitting concerns. Since the data is similar to the original data, we expect higher-level features in the DCNN to be relevant to this dataset as well. Hence, the best idea might be to train a linear c