deep learning on computer vision...
Post on 04-Oct-2020
2 Views
Preview:
TRANSCRIPT
d
National Taiwan University
Dept. of EE, Research Assistant
Yen-Cheng Liu
Deep Learning on Computer VisionApplications
Outline
• Convolution Neural Network (CNN)
• Applications• Super-resolution
• Compressed Artifact Reduction
• Reconstruct Compressed Image
• Context Inpainting
• Colorization
• Sketch Simplification• Style Transfer
1
• Depth Estimation
• Semantic Segmentation
• Standard Model
• Image-to-Image Model
Outline
• Convolution Neural Network (CNN)
• Applications• Super-resolution
• Compressed Artifact Reduction
• Reconstruct Compressed Image
• Context Inpainting
• Colorization
• Sketch Simplification• Style Transfer
1
• Depth Estimation
• Semantic Segmentation
• Standard Model
• Image-to-Image Model
2
No Math in this tutorial
No Math in this tutorial
2
How I feel when no math
appear ing in a paper
Convolutional Neural Networkn CNN History
• 1990s, CNN used to be the dominant tool, but then fell out of fashion, particularly in computer
vision, with the rise of support vector machines(SVM).
• In 2012, CNN has become popular again due to significant success on the ILSVRC
n Standard structure
Convolution layers and pooling layers Fully connected layers 3
Convolutional Neural Networkn Component
• Convolution layers, Pooling layers and Fully connected layers
• Purpose: originally for classification (i.e. LeNet)
4
Convolution Layer
Image Credit: Stanford CS231n
Pooling Layer Fully-Connected Layer
Input Feature Map Filters
Output Feature Map
Standard CNN Model
5
Convolution layers and pooling layers Fully connected layers
Input Output
6
What you did yesterday……
Ground Truth Label
+
7
Ground Truth Label
+Human Machine
What you did yesterday……
What you did yesterday……
Convolution layers and pooling layers Fully connected layers
8
Input
“9” “5” “2” ”7”
Digit
Recognition
Output
Standard CNN Model- Example
9
Object Datasets
CNN
Convolution layers and pooling layers Fully connected layers
10
InputOutput
“Dog”
Object
Recognition
Standard CNN Model- Example
Standard CNN Model- Example
11
Face Datasets
CNN
Convolution layers and pooling layers Fully connected layers
12
InputOutput
“柯P”
Face
Recognition
Standard CNN Model- Example
Standard CNN Model- Example
13
某Datasets
+ Label
CNN
某Recognition
Standard CNN Model- Example
某Datasets
+ Label
CNN
某Recognition
13
Standard CNN Model- Example
14
某Datasets
+ Label
CNN
某Recognition
Jonathan Long Evan Shelhamer Trevor Darrell
(from UC Berkley)
Image-to-Image Model
Convolution layers and pooling layers Fully connected layers
15
Input Output
Image-to-Image Model
Convolution layers Fully connected layers
15
Input Output
Image-to-Image Model
Convolution layers Fully connected layers
15
Input Output
Input
Image-to-Image Model
Convolution layers Fully connected layers
Input Output
Image
Input
Image
OutputUp-sampling
15
Image-to-Image Model
Convolution layers Fully connected layers
Input Output
Image
Input
Image
Output
15
Image-to-Image Model
Convolution layers Fully connected layers
Input Output
Input Output
15
Applications
16
Super-resolution
17
Image Credit: Wei-Sheng Lai @ UC Merced
Super-resolution
18
Super-resolution- Input : Low-resolution image Y
- Output : High-resolution image F(Y)
Image Credit: Wei-Sheng Lai @ UC Merced
[1] “Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution”, Lai et al., CVPR ‘17
[2] “Image super-resolution using deep convolutional networks”, Dong et al., TPAMI ‘17
[3] “Accelerating the super-resolution convolutional neural network”, Dong et al., ECCV ‘16
[5] “Deeply-recursive convolutional network for image super-resolution”, Kim et al., CVPR ‘16 [4] “Accurate image super-resolution using very deep convolutional network”, Kim et al., CVPR ‘16
[1]
[2]
[3]
[4]
[5]
19
[6] Learning a Deep Convolutional Network for Image Super-Resolution, Dong et al., ECCV ‘14
Super-resolution
20
Reconstructing Compressed Image• Kulkarni et al.[4] present a non-iterative and extremely fast algorithm to
reconstruct images from compressively sensed (CS) random measurements
21[7] “ReconNet: Non-Iterative Reconstruction of Images from Compressively Sensed Random Measurements”, Kulkarni et al., CVPR ‘16
Reconstructing Compressed Image
22[7] “ReconNet: Non-Iterative Reconstruction of Images from Compressively Sensed Random Measurements”, Kulkarni et al., CVPR ‘16
Image Artifacts Removal• Compressed artifacts can be removed by using ARCNN [8]
23[8] “Compression Artifacts Reduction by a Deep Convolutional Network”, Dong et al., ICCV ‘15
Image Artifacts Removal
[8] “Compression Artifacts Reduction by a Deep Convolutional Network”, Dong et al., ICCV ‘15 24
Context Inpainting• Pathak et al.[9] generate the contexts of an arbitrary image region
conditioned on its surroundings using CNN
25[9] ”Context Encoders: Feature Learning by Inpainting”, Pathak et al., CVPR ‘16.
Context Inpainting• Pathak et al.[9] generate the contexts of an arbitrary image region
conditioned on its surroundings using Generative Adversarial Net
26[9] ”Context Encoders: Feature Learning by Inpainting”, Pathak et al., CVPR ‘16.
Context Inpainting
27[9] ”Context Encoders: Feature Learning by Inpainting”, Pathak et al., CVPR ‘16.
Face Inpainting• Input: Corrupted Facial Image• Output: Complete Facial Image
28[10] ”Generative Face Completion”, Li et al., CVPR ‘17.
[11] “DeMeshNet: Blind Face Inpainting for Deep MeshFace Verification”, Zhang et al., CVPR ‘17
OutputInput Input Output
Face Rotation• Input: Facial Image• Output: Facial Image with given angle
29[12] ” Rotating Your Face Using Multi-task Deep Neural Network ”,Yim et al., CVPR ‘15.
[13] “Disentangled Representation Learning GAN for Pose-Invariant Face Recognition”, Trum et al., CVPR ‘17
Attribute Manipulation
30[14] StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation
Colorization
31
Colorization• Cheng et al.[14] investigates into the colorization problem which converts a
grayscale image to a colorful one
32[15] “Deep Colorization”, Cheng et al., ICCV ‘15.
Colorization• Cheng et al.[14] investigates into the colorization problem which converts a
grayscale image to a colorful version
33[15] “Deep Colorization”, Cheng et al., ICCV ‘15.
Colorization• Satoshi et al.[16] propose a technique to automatically colorize grayscale
images that combines both global priors and local image features .
Global image priors are extracted from entire image
Local image feature are computed from small image pattern
[16] “Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with SimultaneousClassification”, Iizuka et al., SIGGRAPH ‘16
34
Colorization• Satoshi et al.[16] propose a technique to automatically colorize grayscale
images that combines both global priors and local image features .
[16] “Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with SimultaneousClassification”, Iizuka et al., SIGGRAPH ‘16
35
Sketch Simplification• Simo-Serra et al.[17] propose CNN structure to simplify sketch drawings
• This architecture can process any resolution due to Fully Convolutional
Neural Network
Input Image Output Image[17] Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup, Simo-Serra et al., SIGGRAPH ‘16 36
Sketch Simplification• Simo-Serra et al.[17] propose CNN structure to simplify sketch drawings
More challenging input rough raster image, instead of vector image
[17] Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup, Simo-Serra et al., SIGGRAPH ‘16 37
Sketch-to-Photo Inversion
[18] “Scribbler: Controlling Deep Image Synthesis with Sketch and Color”, Sangkloy et al. , CVPR 2017
Input
Output
38
Attribute Manipulation + Style Transfer
[19] “Detach and Adapt: Learning Cross-Domain Disentangled Deep Representation”, CVPR ’1839
No Label Supervision
Input
Output
Outline
• Convolution Neural Network (CNN)
• Applications• Super-resolution
• Compressed Artifact Reduction
• Reconstruct Compressed Image
• Context Inpainting
• Colorization
• Sketch Simplification• Style Transfer
40
• Depth Estimation
• Semantic Segmentation
• Standard Model
• Image-to-Image Model
Outline
• Convolution Neural Network (CNN)
• Applications• Super-resolution
• Compressed Artifact Reduction
• Reconstruct Compressed Image
• Context Inpainting
• Colorization
• Sketch Simplification• Style Transfer
• Depth Estimation
• Semantic Segmentation
• Standard Model
• Image-to-Image Model
40
• Gatys et al.[20] propose a system which use neural representation to
separate and recombine content and style of arbitrary images
Artistic Image Style Transfer
[20] “A Neural Algorithm of Artistic Style”, Gatys et al., CVPR ‘1641
Artistic Image Style Transfer
• Based on VGG-19 framework
• Extract the feature map of single photo and artwork to generate the image which mixcontent and style
VGG-19
Content / Style
Representation
[20] “A Neural Algorithm of Artistic Style”, Gatys et al., CVPR ‘1642
Artistic Image Style Transfer
• Based on VGG-19 framework
• Extract the feature map of single photo and artwork to generate the image which mixcontent and style
43[20] “A Neural Algorithm of Artistic Style”, Gatys et al., CVPR ‘16
Artistic Image Style Transfer
• Feed-forward CNN Model (1000x faster than Gatys et al.)
44
• Loss Network is based on Pre-trained VGG-19 framework
[21] "Perceptual losses for real-time style transfer and super-resolution.“, Johnson et al., ECCV ‘16.
Artistic Video Style Transfer
https://www.youtube.com/watch?v=Khuj4ASldmU
45[22] "Artistic style transfer for videos.“, Ruder et al., arXiv ‘16.
Artistic 360 Video Style Transfer
46
https://www.youtube.com/watch?v=pkgMUfNeUCQ
[23] " Artistic style transfer for videos and spherical images.“, Ruder et al., arXiv ‘17.
Deep Photo Enhancer
47[24] "Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs.“, Chen et al., CVPR ‘18
Deep Photo Enhancer
47[24] "Deep Photo Enhancer: Unpaired Learning for Image Enhancement from Photographs with GANs.“, Chen et al., CVPR ‘18
Single Image Depth Estimation
• Depth Estimation CNN– Input: RGB image
– Output: Depth/Disparity Estimation
48[25] “Unsupervised Monocular Depth Estimation with Left-Right Consistency”, Godard et al., CVPR ’17
[26] “Unsupervised Learning of Depth and Ego-motion from Video”, Zhou et al., CVPR ‘17
[27] “Semi-Supervised Deep Learning for Monocular Depth Map Prediction”, Kuznietsov et al., CVPR ‘17
Single Image Depth Estimation
• Depth Estimation CNN– Input: RGB image
– Output: Depth/Disparity Estimation
48
Semantic Segmentation
• Semantic Segmentation CNN (Pixel-wise classification)
– Input: RGB image
– Output: Pixel-wise classes prediction
49[28] “Fully Convolutional Networks for Semantic Segmentation”, Long et al., CVPR ’15
[29] “Pyramid Scene Parsing Network”, Zhao et al., CVPR ‘17
Semantic Segmentation
• Semantic Segmentation CNN (Pixel-wise classification)
– Input: RGB image
– Output: Pixel-wise classes prediction
50
https://www.youtube.com/watch?v=qWl9idsCuLQ
Semantic Segmentation
• Semantic Segmentation CNN (Pixel-wise classification)
– Input: RGB image
– Output: Pixel-wise classes prediction
50
Today’s Practice!
Semantic Segmentation
• Semantic Segmentation CNN (Pixel-wise classification)
– Input: RGB image
– Output: Pixel-wise classes prediction
50
Today’s Practice!
Style Transfer + Semantic Segmentation• Champandard [30] introduce a novel concept to augment artistic style
algorithm with semantic annotation
Doodle by Human Result
[30] Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks, Champandard et al., arXiv, Mar, 201651
Style Transfer + Semantic Segmentation• Champandard [11] introduce a novel concept to augment artistic style
algorithm with semantic annotation
51
Semantic Segmentation
by CNN or HumanPainting
[30] Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks, Champandard et al., arXiv, Mar, 2016
Style Transfer + Semantic Segmentation• Champandard [11] introduce a novel concept to augment artistic style
algorithm with semantic annotation
51
Semantic Segmentation
by CNN or HumanPainting
[30] Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks, Champandard et al., arXiv, Mar, 2016
Summary
52
Today’s Presentation
Sty le Trans fe r
S e m a n t i c S e g m e n t a t i o n
Depth Estimation
Photo EnhancerAttribute Manipulation
S u p e r -Reso lu t ion
Colorization
I m a g e Art i facts R e m o v a l
C o n t e x t I n p a i n t i n g
Summary
53
Computer Vision
Today’s Presentation
54
Conclusion• Research areas including computer vision, image processing and computer graphics
have a great success based on Deep Learning
• Convolution Neural Network is still evolving and continually achieve magical
performance• Unsupervised learning
• Meta learning (Learning to learn)
• Explanation of neural network
55
top related