Yunchao Wei

Weakly Supervised Semantic Segmentation with Image-level Annotation

https://weiyc.github.io

The trend of weakly-supervised learning

2014 2015 2016

CVPR ICCV/ECCV

2013 2014

5

3

8

6

8 8

12 12

Why do we need WSL?

•The success of DCNN-based object recognition approaches rely on alarge number of labeled images

•Labeling a large amount of images is very costly in terms of bothfinance and human effort.

•Object detection

•Semantic Segmentation

Semantic Segmentation

… …

Fully-convolutional Segmentation Network

Loss

Segmentation

Task

•Fully supervised scheme

Semantic Segmentation

•Weakly-supervised scheme with image-level annotation

person

horsetable

images

annotations

…

…

Weakly-supervised

Semantic Segmentation

Test Image

Proposal-based Solution

Learning to segment with image-level annotations. PR 2016

Proposal-based Solution

Hypotheses-CNN-Pooling

HCP: A flexible CNN framework for multi-label image classification Yunchao Wei, etc. TPAMI 2016

Localization Map Generation

•Exhaustedly examine each proposal togenerate localization

•Time consuming

•Introducing false negative pixels(background)

STC: Simple to Complex

Simple Images Complex Images

•Motivation

STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017

STC: Simple to Complex

•Simple images with the corresponding saliency maps

STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017

STC: Simple to Complex

•Framework•Initial-DCNN

•Enhanced-DCNN

•Powerful-DCNN

STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017

STC: Simple to Complex

•Flickr-Clean（40K）

STC: Simple to Complex

Networks Training Set mIoU

I-DCNN Flickr-Clean 44.1

E-DCNN Flickr-Clean 46.8

P-DCNN Flickr-Clean+VOC 49.8

Ablation Analysis on Pascal VOC12 val

Comparisons on Pascal VOC12 test

Methods mIoU

MIL-FCN (ICLR 2015) 24.9

CCNN (ICCV 2015) 35.5

EM-Adapt (ICCV 2015) 39.6

MIL-ILP-Seg (CVPR 2015) 40.6

STC (ours) 51.2

STC: A simple to complex framework for weakly-supervised semantic segmentation TPAMI 2017

STC: Simple to Complex

•Testing Results

•Shortcomings•Depend on a large number of simple images for training.

Image Result GT Image Result GT

Object Region Mining with AE

•Top-down attention• Class Activation Mapping [1]; Excitation Backpropagation [2]

[1] Learning Deep Features for Discriminative Localization. Bolei Zhou etc. CVPR 2016

Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)

[2] Top-down Neural Attention by Excitation Backprop. Jianmin Zhang etc. ECCV 2016

CVPR 2016 ECCV 2016

Object Region Mining with AE

•Motivation•Classification networks are only responsive to small and sparse discriminative regions from object of interest

•How to obtain dense and integral object-related regions for learning to semantic segmentation?

Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)

Object Region Mining with AE

•Solution: Adversarial erasing (AE)

Some visualized samples

Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)

Object Region Mining with AE

•Framework of AE

Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)

Object Region Mining with AE

•Examples of mined object regions produced by the AE approach

Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)

Object Region Mining with AE

•Online prohibitive segmentation learning (PSL) for Semantic Segmentation

PSL

Producing Segmentation Mask

Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)

Object Region Mining with AE

•Ablation Analysis on Pascal VOC12 val

0 150

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Epoch

Loss

AE-step4

AE-step3

AE-step2

AE-step1

AE-Steps mIoU

AE-step1 44.9

AE-step2 49.5

AE-step3 50.9

AE-step4 48.8

Training Schemes mIoU

w/o PSL 50.9

w/ PSL 54.1

w/ PSL+ 55.0

Adversarial erasing

Prohibitive Segmentation Learning

Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)

Object Region Mining with AE

•Comparisons on Pascal VOC12 testMethods mIoU

MIL-FCN (ICLR 2015) 24.9

CCNN (ICCV 2015) 35.5

EM-Adapt (ICCV 2015) 39.6

MIL-ILP-Seg (CVPR 2015) 40.6

STC (PAMI 2016) 51.2

DCSM (ECCV 2016) 45.1

BFBP (ECCV 2016) 48.0

SEC (ECCV 2016) 51.7

AF-SS(ECCV 2016) 52.7

AE-PSL (ours) 55.7

images predictions GT

Object region mining with adversarial erasing: A simple classification to semantic segmentation approach CVPR2017 (oral)

Future work

22

•Simultaneous weakly-supervised object detection and semanticsegmentation

•Semi-supervised object detection and semantic segmentation

23