local invariant feature descriptors bin fan national laboratory of pattern recognition (nlpr)...
TRANSCRIPT
Local Invariant Feature Descriptors
Bin Fan
National Laboratory of Pattern Recognition (NLPR)Institute of Automation, Chinese Academy of
Sciences
局部图像特征描述 —— 应用
• Wide-Baseline Image Matching– Structure from Motion, Image-based Localization,
Image Stitch
• Object/Instance/Scene Recognition• Object Detection• Image Retrieval
Structure From Motion
Object Recognition
Database
…
Design method:• Handcrafted Descriptors• Data-driven Descriptors
Categories of Descriptors
Developments: Handcrafted Descriptors
• 1999, SIFT [Citation: 23819]• 2003, Shape Context• 2006, SURF [Citation: 4093]• 2008, SMD, DAISY• 2009, OSID, CS-LBP• 2010, BRIEF, HRI-CSLTP, BiCE• 2011, ORB, BRISK, LIOP, MROGH• 2012, FREAK, KAZE, SYM• 2013, Line Context
• 2004, PCA-SIFT• 2007, LDE, Learning descriptor[Brown et al.]• 2009, Best DAISY• 2012, D-BRIEF, Learning descriptor by convex
optimization[Simonyan et al.], BGM/LBGM, LDAHash
• 2013, BinBoost, SQ-SIFT/DAISY
Developments: Data-driven Descriptors
Design method:• Handcrafted Descriptors• Data-driven Descriptors
Encode information:• Gradient-based Descriptors• Intensity-based Descriptors• Descriptor-based Descriptors
Categories of Descriptors
• Gradient-Based– SIFT、 DAISY、 BiCE、MROGH、 BGM、
LBGM、 BinBoost、 Learning Descriptor[Brown et al., Simonyan et al.]
• Intensity-Based– CS-
LBP、 OSID、 BRIEF、 ORB、 BRISK、 FREAK、 LDE、 D-BREIF、 LIOP
• Descriptor-Based– LDAHash, LDP[Cai et al.,PAMI’11]
Design method:• Handcrafted Descriptors• Data-driven Descriptors
Data type:• Floating-point Descriptors• Binary Descriptors
Encode information:• Gradient-based Descriptors• Intensity-based Descriptors• Descriptor-based Descriptors
Categories of Descriptors
• Floating-point Descriptors– SIFT、 SURF、 DAISY、 CS-LBP、 OSID、
MROGH、 LIOP、 LBGM、 LDE…
• Binary Descriptors– BiCE、 BRIEF、 ORB、 FREAK、 BRISK、
BGM、 BinBoost、 LDAHash、 D-BRIEF…
Name Mem. Com. Mat. Dist. Rob.
Floating point descriptors
SIFT ●●● ●●● ●●● ●●● ●●●
SURF ●○ ●● ●○ ●●○ ●●○
DAISY ●●●●○ ●●●● ●●●●○ ●●●○ ●●●○
LIOP ●●●○ ●●● ●●●○ ●●●● ●●●●
MROGH ●●●● ●●●●○ ●●●● ●●●●○ ●●●●○
Binary descriptors
BRIEF ●○ ○ ● ●●● ●●
ORB ●○ ● ● ●●● ●●○
FREAK ●○ ● ● ●●● ●●○
D-BRIEF ○ ○ ○ ●● ●●
BinBoost ● ●● ○ ●●● ●●●
Handcrafted Descriptors - SIFT
• Binning of Spatial Coordinates and Gradient Orientations• Soft Assignment of Binning• 4x4 spatial grids, 8 gradient orientations, 128 dim SIFT• Normalization
SIFT Descriptor [Lowe’99]
Handcrafted Descriptors - DAISY
• Log-polar grid arrangement• Gaussian pooling of histograms of gradient orientations• Efficient for dense computation, but not for sparse keypoints!
DAISY Descriptor [Tola et al.’08]
Descriptor Learning – Data Driven Methods
Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12]
Normalized Patch
SmoothLow-level
feature extraction
Spatial pooling
Post process
Projection
Descriptor
Learning
Descriptor Learning – Data Driven Methods
Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12]
• Pre-defined low level features: gradient-based, filter bank based• Pre-defined spatial poolings: SIFT-like, DAISY-like, GLOH-like• Optimized combination of low level feature + spatial pooling• Projection: PCA, LDE …
1st: DAISY-like spatial pooling + filter bank [high Dim]2nd: DAISY-like spatial pooling + gradient [moderate Dim]
PCA is better than LDE for projecting descriptor
Descriptor Learning – Data Driven Methods
Simonyan et al.’s method [ECCV’12]
Normalized Patch
SmoothGradient
map calculation
Spatial pooling
Projection
Descriptor
Learning
• Spatial pooling is constrained to rings• Using L1 regularization to select pooling rings from a large pool• Max-Margin based objective function [convex]• Best reported results in the Brown et al.’s dataset
Handcrafted Binary Descriptors
Pioneering work: LBP
Handcrafted Binary Descriptors
1 1( ) ( ; , ), , ( ; , ) {0,1}nn n nf P P x y P x y
Construct descriptor by binary tests:
Binary tests:
1, ( ) ( )( ; , )
0, ( ) ( )
P x P yP x y
P x P y
Pre-defined positions for binary tests:2 2
(0, ), (0, )25 25
S Sx G y G
BRIEF [ECCV’10, PAMI’12]
Handcrafted Binary Descriptors - BRIEF
Low memory, Fast to compute and match
Limited performance
Handcrafted Binary Descriptors
FREAK [CVPR’12]
Organizing sampling points analogous to retina structure
Learning Binary Descriptors
D-BRIEF [ECCV’12]
• Linear representation of projection matrix by Box/Gaussian/Rect filters• Approximate projection by filter responses• Efficient computation of Box/Gaussian/Rect filter responses• Binarization after discriminative projection• Extremely compact [only 32bits = 4 bytes]
Learning Binary Descriptors
BGM [NIPS’12]
• Explore gradient orientation maps as weak learners• Each bit is construct by one weak learner• Select discriminative gradient orientation maps by boosting
(P1(1), P2
(1),c(1)) (P1(2), P2
(2),c(2)) (P1(n), P2
(n),c(n))
…
Learning Binary Descriptors
BinBoost [CVPR’13]
• Each bit as a linear combination of many gradient orientation maps• Optimization based on boosting• Very compact [64 bits = 8 bytes]
Dataset and Evaluation
• Different contexts Image Matching Object/Instance Recognition Image Retrieval
Dataset and Evaluation: Matching
…
Oxford dataset [2D scenes]: popular benchmarkhttp://www.robots.ox.ac.uk/~vgg/research/affine/index.htmlK. Mikolajczyk, C. Schmid, A performance evaluation of local descriptors. PAMI’05
Dataset and Evaluation: Matching
Oxford dataset [2D scenes]: popular benchmark
Evaluation protocol: recall vs. 1-precision
#
#
correct matchesrecall
correspondences
#
#
correct matchesprecision
matches
Dataset and Evaluation: Matching
Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptorshttp://www.cs.ubc.ca/~mbrown/patchdata/patchdata.htmlM. Brown, G. Hua and S. Winder, Discriminant Learning of Local Image Descriptors. PAMI’12
Three different subsets, each of which has more than 400k patch pairs
Liberty Notre Dame Yosemite
Dataset and Evaluation: Matching
Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptors
Evaluation protocol: False Positive Rate(FPR) vs. Recall
# False PositivesFPR
# Negatives
#True PositivesRecall
# Positives
Dataset and Evaluation: Recognition
• Dataset: Ukbench, ZuBuD, …• Evaluation Protocol: Recognition rate, recall
Dataset and Evaluation: Retrieval
• Dataset: Oxford/Paris Building, Holidays• Evaluation Protocol: mAP, Precision vs. Recall
AP(Average Precision): Precision across all recallsmAP: mean AP of all queries
Resources
• OpenCV: http://opencv.org/– SIFT, SURF, BRISK, BRIEF, ORB, FREAK
• VLFeat: http://www.vlfeat.org/– SIFT, LIOP, Covariant Feature Detectors
• Oxford VGG: http://www.robots.ox.ac.uk/~vgg/research/affine/index.html
• Authors’ pages…
Published Evaluations: Matching
1. K. Mikolajczyk and C. Schmid, A Performance Evaluation of Local Descriptors. PAMI’05
2. P. Moreels and P. Perona, Evaluation of Features Detectors and Descriptors based on 3D objects. IJCV’07
3. Anders Lindbjerg Dahl et al., Finding the Best Feature Detector-Descriptor Combination. 3DIMPVT’11
4. O.Miksik and K. Mikolajczyk, Evaluation of Local Detectors and Descriptors for Fast Feature Matching, ICPR’12
5. J. Heinly et al., Comparative Evaluation of Binary Features, ECCV’12
Published Evaluations: Classification/Recognition
1. K. Mikolajczyk et al., Local Features for Object Class Recognition. ICCV’05
2. E. Seemann et al., An Evaluation of Local Shape-Based Features for Pedestrian Detection. BMVC’05
3. M. Stark and B. Schiele, How Good are Local Features for Classes of Geometric Objects. ICCV’07
4. J. Zhang et al., Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, IJCV’07
5. K. E. A. Van de Sande et al., Evaluation of Color Descriptors for Object and Scene Recognition, PAMI’10
Joint work with Zhenhua Wang
Our Work
1. Feature Description by Intensity Order Pooling2. Local Intensity Order Pattern
Feature Description by Intensity Order Pooling
With a reference orientation:SIFT, SURF, DAISY, CS-LBP …
Category of handcrafted descriptors
With a reference orientation:SIFT, SURF, DAISY, CS-LBP …
Category of handcrafted descriptors
+: encode spatial information, high discriminability-: sensitive to orientation estimation error
36%
64%
Match vs. Orientation error
With a reference orientation:SIFT, SURF, DAISY, CS-LBP …
Category of handcrafted descriptors
RobustnessDistinctiveness
+: encode spatial information, high discriminability-: sensitive to orientation estimation error
Without a reference orientation:RIFT, Spin image
Category of handcrafted descriptors
+: inherently rotation invariance, robust to orientation estimation error-: discard some spatial information, limited discriminability
0 255/2π
0
r
Without a reference orientation:RIFT, Spin image
Category of handcrafted descriptors
RobustnessDistinctiveness
+: inherently rotation invariance, robust to orientation estimation error-: discard some spatial information, limited discriminablity
With a reference orientation:SIFT, SURF, DAISY, CS-LBP …
Without a reference orientation:RIFT, Spin image
Category of handcrafted descriptors
RobustnessDistinctiveness
+: encode spatial information, high discriminability-: sensitive to orientation estimation error
+: inherently rotation invariance, robust to orientation estimation error-: discard some spatial information, limited discriminablity
With a reference orientation:SIFT, SURF, DAISY, CS-LBP …
Without a reference orientation:RIFT, Spin image
Category of handcrafted descriptors
RobustnessDistinctiveness
+: encode spatial information, high discriminability-: sensitive to orientation estimation error
+: inherently rotation invariance, robust to orientation estimation error-: discard some spatial information, limited discriminablity
Robustness
Distinctiveness
Our Solution
Gradient orientation maps [SIFT]
Center-symmetrical binary pattern [CS-LBP]
Construct a local coordinate for low-level feature computation
……
……
…
……
Our Solution
Pool low-level features by intensity orders
Using multiple support regions
Our Solution
Using multiple support regions
Gradient orientation maps -> MROGH
Center-symmetrical binary pattern -> MRRID
Code: http://www.openpr.org.cn/index.php/89-MROGH-v1.1/View-details.html
Multiple Support Regions vs. Single Support Region
Experiments
MROGH MRRID SIFT
SR-i: Results of using the i-th support regionMR: Results of using multiple support region
Averaged results over 140 image pairs
Experiments
Image Matching – Oxford Dataset
Hessian-Affine, Viewpoint change
Experiments
Image Matching – Oxford Dataset
Harris-Affine, Image Blur
Experiments
Object Recognition:
Datasets: 53 Objects, ZuBuD, Ukbench
265 images of 53 objectsEach object has 5 images of different viewpoints
Experiments
Object Recognition:
Datasets: 53 Objects, ZuBuD, Ukbench
1005 images of 201 buildings in the Zurich cityEach building has 5 images of different viewpoints, across seasons
Experiments
Object Recognition:
Datasets: 53 Objects, ZuBuD, Ukbench
10200 images of 2550 objects [first 4000 images used here]Each object has 4 images of different viewpoints
Experiments
Object Recognition:
53 ObjectsRIFT SIFT DAISY MROGH MRRID
37.0% 52.2% 61.2% 72.5% 57.4%
ZuBuDRIFT SIFT DAISY MROGH MRRID
66.8% 75.5% 83.1% 88.1% 78.6%
UkbenchRIFT SIFT DAISY MROGH MRRID
34.0% 48.2% 58.3% 74.0% 57.5%
Experiments
Recognition examples: 53 Objects
input images
Experiments
Recognition examples: ZuBuD
input images
Experiments
Recognition examples: Ukbench
input images
Local Intensity Order Pattern
Local Intensity Order Pattern• Explore the relative intensity relationship among
neighboring points• Rotationally invariant computation of neighboring points’
intensities• Intensity order based pooling
Code: http://vision.ia.ac.cn/Students/wzh/publication/liop/index.html http://www.vlfeat.org/api/liop.html
Experiments
Image Matching: Oxford dataset
Experiments
Image Matching: Oxford dataset
Experiments
Image Matching: Complex Brightness Change
Questions?
Thank you