miru2014 tutorial deeplearning

Download MIRU2014 tutorial deeplearning

Post on 09-Dec-2014

13.463 views

Category:

Engineering

0 download

Embed Size (px)

DESCRIPTION

Tutorial material of Deep Learning in MIRU2014.

TRANSCRIPT

  • 1. Deep Learning

2. Deep Learning(2011)(7F. Seide, G. Li and D. Yu, Conversational Speech Transcription UsingContext-Dependent Deep Neural Networks., INTERSPEECH2011.(2012)CNNA. Krizhevsky, I. Sutskever and G. E. Hinton. "ImageNet Classificationwith Deep Convolutional Neural Networks." NIPS. Vol. 1. No. 2. 2012.2 3. Deep Learning&3 4. Deep LearningITFacebook(AI Lab)LeCunRanzatoGoogleDNNResearch(HintonDeep MindYahoo IQEngine()LookFlowBaidu4 5. Deep Learning641711142010 2011 2012 2013 2014ICCVCVPR205ICPRLSVRC2012(ICCV workshop)512ECCV 6. Deep Learning(20122013)Large Scale Visual Recognition Challenge 20122013Deep Learning+ (ReLu, dropout1000LSVRC2012WEBhttp://image-net.org/challenges/LSVRC/2012/ilsvrc2012.pdfhttp://image-net.org/challenges/LSVRC/2012/supervision.pdf6 7. Deep Learning(2012)SuperpixelC.Farabet, C.Couprie, L.Najman, Y.LeCun, Learning Hierarchical Features for Scene Labeling., PAMI2012. 7 8. Deep Learning(2012)Deep learningJ.Xie, L.Xu, E.Chen, Image Denoising and Inpainting with Deep Neural Networks, NIPS2012.8 9. Deep Learning(2013)INRIA9x9P.Sermanet, K.Kavukcuoglu, S.Chintala, Y.LeCun, Pedestrian Detection with Unsupervised Multi-Stage Feature Learning,CVPR2013.9 10. 10 11. Deep Learning(2014)NVIDIATegra K116-17fps140x60I. Sato, H. Niihara, Beyond Pedestrian Detection: Deep Neural Networks Level-Up Automotive Safety,201411 12. Deep LearningDeep Belief Object Recognition (iOS)(2014)iOS00msKrizhevskySDKhttps://www.jetpac.com/12 13. Project Adamhttps://www.youtube.com/watch?feature=player_embedded&v=zOPIvC0MlA413 14. Deep LearningHinton (): https://www.cs.toronto.edu/~hinton/Auto encoderdrop outDeep LearningLeCun(CNNhttp://yann.lecun.comDeep Learning(CNN):Ranzato(Facebook)Kavukcuoglu(DeepMind)*FacebookAI Lab14 15. Deep LearningSchmidhuber (IDSIA)Competitionhttp://www.idsia.ch/~juergen/Deep LearningGPUX.Wang ( )http://mmlab.ie.cuhk.edu.hk/project_deep_learning.html15 16. Deep LearningDeep LearningRestrictedBoltzmannMachinesDeep BeliefNetworksMaxpoolingDeepBoltzmannMachinesConvolutionalNeuralNetworksDeep NeuralNetworksBack-propagationContrastiveDivergenceDropoutMaxoutDropconnect16 17. Deep LearningRestrictedBoltzmannMachinesDeep BeliefNetworksMaxpoolingDeepBoltzmannMachinesConvolutionalNeuralNetworksDeep NeuralNetworksBack-propagationContrastiveDivergenceDropoutMaxoutDropconnect17 18. Deep LearningMulti-LayerPerceptronDeep LearningRestrictedBoltzmannMachinesDeep BeliefNetworksConvolutionalNeuralNetworksDeepBoltzmannMachinesDeep NeuralNetworksMax poolingMaxoutDropoutDropconnect18 19. MLPRBMMulti-Layer Perceptron(MLP) Restricted Boltzmann Machine(RBM)m )p(xi =1|Y ) =s ( wijyj + aij=1n )p(yj =1| X) =s ( wijxi + bji=1m )yi =s ( wijxj + bjj=119 20. DNNDBNDeep Neural Networks(DNNs) Deep Belief Networks(DBN)(Back propagation)(Contrastive Divergence)+(Back propagation)20 21. Convolutional Neural NetworksConvolutional Neural Networks21 22. Convolutional Neural NetworksY. LeCun, et.al. Gradient-based Learning Applied to Document Recognition, Proc. of The IEEE, 1998. 22 23. Convolutional Neural NetworksConvolutional Neural Networks23 24. Convolution Layer()Input image 10x10 kernel 3x3 Feature map 8xConvolutionResponsef8Activationfunction24Convolutions 25. Convolution Layer()25ActivationInput image 10x10 function Feature map 8x8ConvolutionResponsekernel 3x3fffConvolutions 26. Activation FunctionRectified Linear Unit(ReLU) Maxout1ReLUf (xi ) =max(xj f (x ,0) i ) =1+ e-x j26Convolutions 27. MaxoutInput imageFeature map10x10kernel3x38x8x3ConvolutionFeature map8x8I.J.Goodfellow, D.Warde-Farley, M.Mirza, A.Courville, and Y.Bengio, Maxout networks.,arXiv preprint arXiv:1302.4389, 2013. 27 28. Pooling LayerFeature mapMax pooling2x2Average pooling2x2Lp poolingmnf (xi ) = ( I(i, j)p *G(i, j))1pi=1j=128 Sampling 29. Fully connection layerx1x2x3xih1h2hjactivation functionhj = f (WT x +bj )29 Fullconnectionw11w12w21w1 jw22 w31w32w3 jwi2wijwi1 30. Classification LayerSoftmaxP(y1)P(y2)P(yM)Mx1x2x3xih1h2hM30classificationP(yi ) =exp(hi )exp(hj )j=1 31. Convolutional Neural NetworksConvolutional Neural Networks31 32. Layer(Stochastic Gradient Descent)32 33. 33Input:x:y(I1,y1),, (xn,yn)ConvolutionFull connectionClassification 34. Input:xi:yi(x1,y1),, (xn,yn)Wyiyi34ConvolutionFull connection Classificationy' = F(W, x)nE = Loss(F(W, xi ), yi )iEWW W -gW 35. mini batchmini batch(SGD)m1m2mkmmkm135n 36. 36 37. Convolutional Neural NetworksConvolutional Neural Networks37 38. 38Input:x:y(x1,y1),, (xn,yn) 39. 39Input:x:y(x1,y1),, (xn,yn) 40. 40Input:x:y(x1,y1),, (xn,yn) 41. Auto encoderInput layerhidden layerReconstruction errorReconstruction layerWTied weightsw1, b1, b141x xh(x) 42. Auto encoderInput layerhidden layerReconstruction errorReconstruction layerWTied weightsw1, b1, b142x xh(x) 43. Auto encoderInput layerhidden layerReconstruction errorReconstruction layerWTied weightsw1, b1, b143x xh(x) 44. Auto encoderInput layerhidden layerReconstruction errorReconstruction layerWTied weightsw1, b1, b144x xh(x) 45. Auto encoderConvolutionalKernelk1k1k1Reconstruction errork, k, b, b45k1feature map 46. Stacked Auto encoderx xInput layerhidden layerReconstruction layerReconstruction errorw2, b2, b2Y.Bengio, P. Lamblin, D. Popovici and H. Larochelle, Greedy Layer-Wise Training of Deep Networks, NIPS0746h(x) 47. Stacked Auto encoderx xInput layerhidden layerReconstruction layerReconstruction errorw2, b2, b247h(x)Y.Bengio, P. Lamblin, D. Popovici and H. Larochelle, Greedy Layer-Wise Training of Deep Networks, NIPS07 48. 48 49. Dropout()Input layerK1KnKernelFully connected layer()mini-batchG. Hinton, N.Srivastava, A.Krizhevsky, I.Sutskever, and R.Salakhutdinov, Improving neural networks bypreventing co-adaptation of feature detectors., arXiv preprint arXiv:1207.0580, 2012. 49 50. Dropconnect()Input layerK1KnKernelFully connected layer()mini-batchL.Wan, M.Zeiler, S.Zhang, Y.LeCun, R.Fergus, Regularization of Neural Network using DropConnect, ICML2013 50 51. Dropout vs DropconnectL.Wan, M.Zeiler, S.Zhang, Y.LeCun, R.Fergus, Regularization of Neural Network using DropConnect, ICML2013 51 52. Data AugmentationElastic Distortion52P.Y. Simard, D. Steinkraus, and J.C. Platt, Best practices for convolutional neural networks applied to visualdocument analysis., ICDAR2003. 53. Global Contrast Normalization53 54. Global Contrast Normalizationpylearn254 55. ZCA whiteningX' =WXWhttp://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf55 56. ZCA whitening56ZCA whitening onlyGlobal contrast normalization +ZCA whitening 57. Normalize Layeractivation function Pooling layer Convolutional layerConvolutional layer Normalize layerPooling layerNormalize layerpooling layerNormalize layer57 58. Normalize LayerLocal contrast normalizationConvolutional layer Normalize layervj,k = xj,k - wp,qxj+p,k+q wp,q =1yj,k =vj,kmax(C,s jk )2 s jk = wpqvj+p,k+qK. Jarrett, K. Kavukcuoglu, M. Ranzato and Y.LeCun ,What is the Best Multi-Stage Architecture forObject Recognition?, ICCV2009 58 59. Normalize LayerLocal response normalizationConvolutional layer Normalize layeryii+N/2j,k = (1+a (ylj,k )2 )bl=i-N/259G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever and R. R. Salakhutdinov ,Improving neuralnetworks by preventing co-adaptation of feature detectors , arxiv2012 60. 60 61. Activation functionsigmoidmaxoutReLUPoolingmax, average, L2 etc.MaxoutFully connectionFully connectionDropout61 62. mini batchAuto encoder62 63. CIFAR10airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truckdata augmentation 85% 63 64. 64Convolutional layer1Classification layerxConvolution layerNormalize layerPooling layerConvolution layerNormalize layerPooling layerClassification layerConvolution layerNormalize layerPooling layer 65. 65Convolutional layer1Classification layerxConvolution layerNormalize layerPooling layerConvolution layerNormalize layerPooling layerClassification layerConvolution layerNormalize layerPooling layerPoolingNormalize layerfully connection layerdropout 66. CIFAR1010.80.60.40.20Conv3Full0Conv3Full1Conv3Full2Conv3Full30 100000 200000 30000066 67. CIFAR1010.80.60.40.20GCN+ZCAZCA0 100000 200000 30000067 68. Convolution layeroriginal image ZCA image GCN + ZCA image68 69. CIFAR10Convolutional Layer10.80.60.40.203x3x25x5x43x4x57x6x40 100000 200000 30000069 70. CIFAR10Convolutional Layer10.80.60.40.204x4x48x8x816x16x1632x32x3264x64x64128x128x128256x256x2560 100000 200000 30000070 71. CIFAR10Convolutional Layer10.80.60.40.204x8x168x16x3216x32x6432x64x12864x128x256128x256x512256x512x1024256x256x2560 100000 200000 30000071 72. Convolutional layer72070001000)x 73. CIFAR10Activation Function10.80.60.40.20maxoutsigmoidReLUTanh0 100000 200000 30000073 74. CIFAR10DropoutAuto Encoder10.80.60.40.20dropoutdropout0 100000 200000 30000074 75. CIFAR10DropoutAuto Encoder10.80.60.40.200.10.010.0010.00010 100000 200000 30000075 76. CIFAR107610.80.60.40.20batch size :5batch size :10batch size :20batch size :25batch size :50batch size :100batch size :1250 100000 200000 77. CIFAR10Normalize Layer10.80.60.40.200 100000 200000 30000077 78. CIFAR103convolution + 1classificationGCN+ZCAconvolution layer128x256x512, 5x5+5x5+4x4Activation function maxout ReLu)full connectiondropout Normalize layer0.00178 79. CPU V.S. GPU(Layer CPU(Core22.6GHz)GPU(GeForceGT690)27.3ms 11.6ms 2.35451.5ms 29.2ms 15.46486ms 14.8ms 32.84Pre training :0.5Fine tuning :0.01Mini-batch :1079 80. Layerx10.1KBx324KB100 (87000)0.35MB1600(410000)1.6MB40x4080 81. 81 82. http://deeplearning.net/tutorial/intro.html82 83. CNNTheano(python)http://deeplearning.net/software/theano/83 84. cuda-convnetKrizhevskyhttps://code.google.com/p/cuda-convnet/84 85. CaffeCaffeUC (T. Darrell)(Kaggle)https://github.com/UCB-ICSI-Vision-Group/decaf-release/85 86. OverFeatOverFeat(C/C++)ImageNethttp://cilvr.nyu.edu/doku.php?id=software:overfeat:startLeCu

View more >