segmentation based features for wide- baseline multi-view...
TRANSCRIPT
![Page 1: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/1.jpg)
ARMIN MUSTAFA, HANSUNG KIM,
EVREN IMRE AND ADRIAN HILTON
Segmentation based features for wide-
baseline multi -view reconstruction
![Page 2: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/2.jpg)
Contents
• Motivation
• Existing methods
• SFD: Segmentation based feature detector
• Results and Evaluation of feature detectors
• Application to sparse and dense reconstruction
• Conclusion
2
![Page 3: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/3.jpg)
Applications
Film
Broadcast
Surveillance
Object Recognition 3
![Page 4: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/4.jpg)
Key application
Sparse and Dense scene reconstruction
Feature detection &
matching between
pair of views
Multiview
Capture Dense scene
reconstruction
Camera calibration
& Sparse scene
reconstruction
4
![Page 5: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/5.jpg)
Key application
Sparse scene reconstruction
5
(Proposed)
![Page 6: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/6.jpg)
Key application
Dense scene reconstruction
SIFT SFD
6
(Proposed)
![Page 7: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/7.jpg)
Large number of features and matches
Good scene coverage
Improved accuracy
Order of magnitude increase in reconstructed
points.
Why SFD? 7
![Page 8: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/8.jpg)
SFD for dense reconstruction
8
Features Original images Segmented images
Feature matches Sparse reconstruction Dense reconstruction
![Page 9: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/9.jpg)
SFD Algorithm
Over-segmentation:
9
Region boundaries represent lines corresponding to local
maxima of the image function
Watershed segmentation Original image
![Page 10: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/10.jpg)
SFD Algorithm
Feature Detection: Feature Illustration
Feature examples Odzemok segmented image
10
![Page 11: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/11.jpg)
SFD Algorithm
Feature Detection:
11
![Page 12: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/12.jpg)
Evaluation: Datasets
Juggler (6 moving) Odzemok
(6 static, 2 moving)
Cathedral (8 static)
12
![Page 13: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/13.jpg)
Evaluation: Datasets
Valbonne
Merton
Rossendale
13
![Page 14: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/14.jpg)
Evaluation: Features and Matches
Outdoor- Dynamic
14
Matches SFD with Watershed SIFT MSER
![Page 15: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/15.jpg)
Outdoor - Static
15
Matches SFD with Watershed SIFT MSER
Evaluation: Features and Matches
![Page 16: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/16.jpg)
SIFT SFD
Evaluation: Sparse reconstruction
Original
16
261 4084
7211 409
1884 12385
![Page 17: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/17.jpg)
17
Some more results
![Page 18: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/18.jpg)
18
Evaluation: Over-segmentation
SFD: Independent of segmentation technique
![Page 19: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/19.jpg)
Detected features Matches
Evaluation: Over-segmentation
0
2000
4000
6000
8000
10000
12000
Watershed Mean-shift SLIC Watershed Mean-shift SLIC
Merton
Odzemok
19
![Page 20: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/20.jpg)
Detected features Matches
Evaluation: Over-segmentation
0
1000
2000
3000
4000
5000
6000
Watershed Mean-shift SLIC Watershed Mean-shift SLIC
Valbonne
Juggler
20
![Page 21: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/21.jpg)
Evaluation: Matches
0
1000
2000
3000
4000
5000
6000
FAST HARRIS MSER ORB SIFT SURF SFD-WA SFD-MS SFD-SLIC
Odzemok
Merton
Nu
mb
er
of
co
rrect
matc
hes
21
![Page 22: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/22.jpg)
Evaluation: Matches N
um
ber
of
co
rrect
matc
hes
0
500
1000
1500
2000
2500
3000
FAST HARRIS MSER ORB SIFT SURF SFD-WA SFD-MS SFD-SLIC
Valbonne
Juggler
22
![Page 23: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/23.jpg)
Evaluation: Time performance
0
10
20
30
40
50
60
70
FAST HARRIS MSER ORB SIFT SURF SFD-WA SFD-MS SFD-SLIC
Odzemok
Merton
Tim
e p
erf
orm
an
ce i
n m
s
23
![Page 24: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/24.jpg)
Evaluation: Time performance T
ime p
erf
orm
an
ce i
n m
s
0
10
20
30
40
50
60
70
80
90
100
FAST HARRIS MSER ORB SIFT SURF SFD-WA SFD-MS SFD-SLIC
Valbonne
Juggler
24
![Page 25: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/25.jpg)
Mean re-projection error (MRE)
Mean re-projection error (MRE) is calculated for Odzemok dataset for various detectors
N is the number of feature matches
(x,y)
(x`, y`)
3D Ground Truth
Image 1 Image 2
25
![Page 26: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/26.jpg)
Accuracy (MRE) Evaluation of SFD
Feature
Detector
Descriptor RC MRE
SFD SIFT 3717 1.351
SIFT SIFT 1269 1.175
MSER SIFT 119 1.390
FAST BRIEF 121 1.483
Re-projection error of SIFT and SFD-WA for Odzemok
26
![Page 27: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/27.jpg)
Repeatability with camera 1 to all other views (15-120 degree baseline).
Evaluation: Repeatability
![Page 28: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/28.jpg)
Conclusions
Novel feature detector for wide-baseline matching
A comprehensive performance evaluation for feature
matching and time performance
Ground truth accuracy evaluation
Further plans include evaluating the utility of SFD
features in applications such as camera tracking and
object recognition.
28
![Page 29: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/29.jpg)
29
Thank you!
Questions??
![Page 30: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/30.jpg)
Accuracy Evaluation of SFD with Harris
and Uniform Sampling
FD Descriptor Features RC
SFD SIFT 13881 3717
Uniform
Sampling SIFT 12284 33
Harris SIFT 13158 145
• Uniform grid sampling is performed by locating features at points of maximum gradient
magnitude with a 13X13 grid
• Experimented on Odzemok dataset
30
![Page 31: Segmentation based features for wide- baseline multi-view ...cvssp.org/projects/4d/SFD/Armin3DV2015PPT.pdf · EVREN IMRE AND ADRIAN HILTON Segmentation based features for wide-baseline](https://reader035.vdocuments.net/reader035/viewer/2022063009/5fc0ebba83e4c0248c57be01/html5/thumbnails/31.jpg)
Evaluation: Repeatability
Repeatability between adjacent views (15-30 degree baseline)