deep learning for image instance segmentation ----mask r-cnn · object instance segmentation....
TRANSCRIPT
![Page 1: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/1.jpg)
Deep Learning for Image Instance Segmentation----Mask R-CNN
Jianping Fan Dept of Computer Science
UNC-Charlotte
Course Website: http://webpages.uncc.edu/jfan/itcs5152.html
![Page 2: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/2.jpg)
Definition of Image Instance Segmentation
Instance segmentation = object detection + semantic segmentation?
![Page 3: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/3.jpg)
![Page 4: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/4.jpg)
![Page 5: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/5.jpg)
Background Review
![Page 6: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/6.jpg)
R-CNN
• R-CNN [4]: The Region-based CNN (R-CNN)• Replace sliding windows with “selective search” region proposals(Uijilings et
al. IJCV 2013)
• Extract rectangles around regions and resize to 227x227 pixels
• Extract features with fine-tuned CNN (that was initialized with network trained on ImageNet before training)
• Classify last layer of network features with SVM, refine bounding box localization (bbox regression) simultaneously
6Slide credit to Ke-Shuan Cheng
![Page 7: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/7.jpg)
R-CNN
7
• R-CNN: The Region-based CNN (R-CNN)
Slide credit to Ke-Shuan Cheng
Region warping is performed for fixed size features
![Page 8: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/8.jpg)
![Page 9: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/9.jpg)
R-CNN Architecture
![Page 10: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/10.jpg)
Regional Proposal Network (RPN)
● Foreground vs Background ● Bounding Box regression● Feed bounding boxes into Fast RCNN
![Page 11: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/11.jpg)
![Page 12: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/12.jpg)
![Page 13: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/13.jpg)
![Page 14: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/14.jpg)
(0,0)
(0,0)(1,3)
(16,48)
Mapping the center of the receptive fields
![Page 15: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/15.jpg)
● k anchor boxes ○ 3 scales (8,16, 32)○ 3 aspect ratios (.5, 1, 2)○ Stride 16
● WHk anchors
Anchor Boxes
W
H Center(x,y)
Feature Map
No
thin
g to
See here.
![Page 16: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/16.jpg)
Anchor Boxes
![Page 17: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/17.jpg)
Anchor Boxes
![Page 18: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/18.jpg)
RPN(Region Proposal Network) Object vs Not an Object
Anchor
Object = 1 to:a) Anchors with the highest
Intersection-over-Union(IoU)b) IoU > 0.7 with any ground truth
box.Not object = -1
a) If IoU <0.3
![Page 19: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/19.jpg)
RPN
512-d
(x,y)
(Sx,Sy)
Mapping the center of the receptive fields
![Page 20: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/20.jpg)
RPN
SigmoidCrossEntropyLoss SmoothL1Loss
(512 × (2 + 4) × 9) parameters for VGG-16)
512
![Page 21: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/21.jpg)
![Page 22: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/22.jpg)
Multi-task loss:
RPN
Only if p*= 1Hyper parameter =10
Mini batch size =256
Number of Anchor locations
![Page 23: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/23.jpg)
![Page 24: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/24.jpg)
![Page 25: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/25.jpg)
Fast R-CNN Architecture
![Page 26: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/26.jpg)
Fast R-CNN Architecture
![Page 27: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/27.jpg)
Fast R-CNN
• Fast R-CNN [5]• Improvement: It only feed the whole image into CNN only once! Then crop
features instead of image itself.
27Slide credit to Ke-Shuan Cheng
![Page 28: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/28.jpg)
Fast R-CNN
• RoI Pooling• The RoI pooling layer uses max pooling to convert the features inside any valid
region of interest into a small feature map with a fixed spatial extent of H × W. (e.g., 2*2 )
28
0.85
0.97
0.84
0.96
Slide credit to Ke-Shuan Cheng
![Page 29: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/29.jpg)
RoI Pooling in Fast R-CNN
RoI Pooling
Slide credit to Silvio Galesso
![Page 30: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/30.jpg)
For each proposalNMS RoI Pooling
Fully connected layers
softmax
Bbox regression
Fast R-CNN
label
bbox
![Page 31: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/31.jpg)
Region of Interest (RoI):
Fast R-CNN
![Page 32: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/32.jpg)
Region of Interest (RoI):
Fast R-CNN
.74 | .39 | .34
.2 | .16 | .73
.83 | .97 | .88
3X3 RoI pooling
![Page 33: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/33.jpg)
Region of Interest (RoI):
Fast R-CNN
.74 | .39 | .34
.2 | .16 | .73
.83 | .97 | .88
7X7 RoI poolingPer proposal
Only a problem for segmentation
![Page 34: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/34.jpg)
![Page 35: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/35.jpg)
Faster R-CNN
• Faster R-CNN [6]• Improvement: Generate RoI by Region Proposal Network.
35Slide credit to Ke-Shuan Cheng
![Page 36: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/36.jpg)
● Two main parts○ Region Proposal
Network○ Fast R-CNN○ (also this) Pre-
trained network
![Page 37: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/37.jpg)
Comparison
• Compare with 3 model
37Slide credit to Ke-Shuan Cheng
![Page 38: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/38.jpg)
![Page 39: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/39.jpg)
Mask-RCNN
![Page 40: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/40.jpg)
Mask R-CNN• Mask RCNN is a simple, flexible, and general framework for
object instance segmentation.
• Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks on each Region of Interest (RoI), in parallel with the existing branch for classification and bounding box regression.
• Mask R-CNN is simple to trained and adds only a small overhead to Faster R-CNN.
• The mask branch is a small Fully Convolutional Network (FCN) applied to each RoI, predicting a segmentation mask in a pixel-to-pixel manner.
40Slide credit to Ke-Shuan Cheng
![Page 41: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/41.jpg)
![Page 42: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/42.jpg)
![Page 43: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/43.jpg)
Mask R-CNN Architecture
![Page 44: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/44.jpg)
![Page 45: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/45.jpg)
![Page 46: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/46.jpg)
Kaiming He, ICCV 2017 Tutorial
![Page 47: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/47.jpg)
Mask R-CNN
• Mask R-CNN [3] is conceptually simple: Faster R-CNN has two outputs for each candidate object, a class label and a bounding-box offset; to this R-CNN added a third branch that outputs the object mask.
47
Faster R-CNN
3rd Branch Object Mask
Slide credit to Ke-Shuan Cheng
![Page 48: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/48.jpg)
![Page 49: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/49.jpg)
![Page 50: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/50.jpg)
Slide credit to Ross Girshick
![Page 51: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/51.jpg)
![Page 52: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/52.jpg)
![Page 53: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/53.jpg)
Kaiming He, ICCV 2017 Tutorial
![Page 54: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/54.jpg)
Slide credit to T. Kim
![Page 55: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/55.jpg)
Slide credit to T. Kim
![Page 56: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/56.jpg)
Slide credit to T. Kim
![Page 57: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/57.jpg)
Slide credit to T. Kim
![Page 58: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/58.jpg)
Slide credit to T. Kim
![Page 59: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/59.jpg)
Slide credit to T. Kim
![Page 60: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/60.jpg)
Slide credit to T. Kim
![Page 61: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/61.jpg)
Slide credit to T. Kim
![Page 62: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/62.jpg)
Slide credit to T. Kim
![Page 63: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/63.jpg)
Slide credit to T. Kim
![Page 64: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/64.jpg)
Slide credit to T. Kim
![Page 65: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/65.jpg)
Kaiming He, ICCV 2017 Tutorial
![Page 66: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/66.jpg)
Kaiming He, ICCV 2017 Tutorial
![Page 67: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/67.jpg)
Kaiming He, ICCV 2017 Tutorial
![Page 68: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/68.jpg)
Slide credit to Silvio Galesso
RoI Align in Mask R-CNN
![Page 69: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/69.jpg)
Slide credit to Silvio Galesso
RoI Align in Mask R-CNN
RoI Align
![Page 70: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/70.jpg)
Slide credit to Silvio Galesso
RoI Align in Mask R-CNN
RoI Align
![Page 71: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/71.jpg)
Slide credit to T. Kim
![Page 72: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/72.jpg)
Slide credit to T. Kim
![Page 73: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/73.jpg)
Slide credit to T. Kim
![Page 74: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/74.jpg)
Slide credit to T. Kim
![Page 75: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/75.jpg)
Slide credit to T. Kim
![Page 76: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/76.jpg)
Slide credit to T. Kim
![Page 77: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/77.jpg)
Slide credit to T. Kim
![Page 78: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/78.jpg)
Slide credit to T. Kim
![Page 79: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/79.jpg)
Slide credit to T. Kim
![Page 80: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/80.jpg)
Slide credit to T. Kim
![Page 81: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/81.jpg)
Slide credit to T. Kim
![Page 82: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/82.jpg)
![Page 83: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/83.jpg)
![Page 84: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/84.jpg)
![Page 85: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/85.jpg)
![Page 86: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/86.jpg)
Kaiming He, ICCV 2017 Tutorial
![Page 87: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/87.jpg)
Slide credit to Ross Girshick
![Page 88: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/88.jpg)
Slide credit to Ross Girshick
![Page 89: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/89.jpg)
Slide credit to Ross Girshick
![Page 90: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/90.jpg)
Mask R-CNN
• RoI Align is Improves miss-align problems of RoI pooling• RoI Align use bilinear interpolation to generate new feature map.
• Do RoI Pooling with aligned feature map
90
RoI Pooling
Bilinear interpolation
Slide credit to Ke-Shuan Cheng
![Page 91: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/91.jpg)
Kaiming He, ICCV 2017 Tutorial
![Page 92: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/92.jpg)
Kaiming He, ICCV 2017 Tutorial
![Page 93: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/93.jpg)
Kaiming He, ICCV 2017 Tutorial
![Page 94: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/94.jpg)
Kaiming He, ICCV 2017 Tutorial
FPN (Feature Pyramid Network)
![Page 95: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/95.jpg)
![Page 96: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/96.jpg)
![Page 97: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/97.jpg)
Kaiming He, ICCV 2017 Tutorial
![Page 98: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/98.jpg)
Mask R-CNN
• They proposed 2 architecture and compared they for Object
Mask branch
• ResNet
• Branch from last Convolutional layer
• Feature Pyramid Network(FPN)
• Branch from RoI
98Slide credit to Ke-Shuan Cheng
![Page 99: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/99.jpg)
Mask R-CNN
99Slide credit to Ke-Shuan Cheng
![Page 100: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/100.jpg)
Mask R-CNN
• Loss function is defined as below:𝐿 = 𝐿𝑐𝑙𝑠 + 𝐿𝑏𝑜𝑥 + 𝐿𝑚𝑎𝑠𝑘
• 𝐿𝑐𝑙𝑠: Cross-Entropy
• 𝐿𝑏𝑜𝑥: IoU𝐴𝑟𝑒𝑎 𝑜𝑓 𝑂𝑣𝑒𝑟𝐿𝑎𝑝
𝐴𝑟𝑒𝑎 𝑜𝑓 𝑈𝑛𝑖𝑜𝑛• 𝐿𝑚𝑎𝑠𝑘: Cross-Entropy between pixel-to-pixel
100Slide credit to Ke-Shuan Cheng
![Page 101: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/101.jpg)
![Page 102: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/102.jpg)
Mask R-CNN• Result
102Slide credit to Ke-Shuan Cheng
![Page 103: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/103.jpg)
![Page 104: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/104.jpg)
![Page 105: Deep Learning for Image Instance Segmentation ----Mask R-CNN · object instance segmentation. •Mask R-CNN, extends Faster R-CNN by adding a branch for predicting segmentation masks](https://reader033.vdocuments.net/reader033/viewer/2022042922/5f6ca5fa00ef10102b74150b/html5/thumbnails/105.jpg)
References https://arxiv.org/pdf/1506.01497.pdf (Faster R-CNN)https://arxiv.org/pdf/1504.08083.pdf (Fast R-CNN)https://arxiv.org/pdf/1506.06981.pdf (R-CNN minus R)https://koen.me/research/pub/uijlings-ijcv2013-draft.pdf (Selective Search for Object Detection)https://arxiv.org/pdf/1703.06870.pdf (Mask R-CNN)http://host.robots.ox.ac.uk/pascal/VOC/https://www.dropbox.com/s/xtr4yd4i5e0vw8g/iccv15_tutorial_training_rbg.pdf?dl=0http://kaiminghe.com/iccv15tutorial/iccv2015_tutorial_convolutional_feature_maps_kaiminghe.pdfhttps://lovesnowbest.site/2018/02/27/Intro-to-Object-Detection/https://blog.deepsense.ai/region-of-interest-pooling-explained/https://tryolabs.com/blog/2018/01/18/faster-r-cnn-down-the-rabbit-hole-of-modern-object-detection/