sift - the scale invariant feature...
TRANSCRIPT
![Page 1: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/1.jpg)
SIFT - The Scale Invariant
Feature Transform
Distinctive image features from scale-invariant keypoints. David G. Lowe, International Journal of Computer Vision, 60, 2 (2004), pp. 91-110
Presented by Ofir Pele.
Based upon slides from:
- Sebastian Thrun and Jana Košecká
- Neeraj Kumar
![Page 2: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/2.jpg)
Correspondence
n Fundamental to many of the core vision problems
– Recognition
– Motion tracking
– Multiview geometry
n Local features are the key
Images from: M. Brown and D. G. Lowe. Recognising Panoramas. In Proceedings of the
) the International Conference on Computer Vision (ICCV2003
![Page 3: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/3.jpg)
Local Features:
Detectors & Descriptors Detected
Interest Points/Regions
Descriptors
<0 12 31 0 0 23 …>
<5 0 0 11 37 15 …>
<14 21 10 0 3 22 …>
![Page 4: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/4.jpg)
Ideal Interest Points/Regions
n Lots of them
n Repeatable
n Representative orientation/scale
n Fast to extract and match
![Page 5: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/5.jpg)
Detector SIFT Overview
1. Find Scale-Space Extrema
2. Keypoint Localization & Filtering
– Improve keypoints and throw out bad ones
3. Orientation Assignment
– Remove effects of rotation and scale
4. Create descriptor
– Using histograms of orientations
Descriptor
![Page 6: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/6.jpg)
Detector SIFT Overview
1. Find Scale-Space Extrema 2. Keypoint Localization & Filtering
– Improve keypoints and throw out bad ones
3. Orientation Assignment
– Remove effects of rotation and scale
4. Create descriptor
– Using histograms of orientations
Descriptor
![Page 7: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/7.jpg)
Scale Space
n Need to find ‘characteristic scale’ for feature
n Scale-Space: Continuous function of scale σ
– Only reasonable kernel is Gaussian:
yxIyxGyxL DD ,*,,,,
[Koenderink 1984, Lindeberg 1994]
![Page 8: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/8.jpg)
Scale Selection
n Experimentally, Maxima of Laplacian-of-Gaussian gives
best notion of scale:
n Thus use Laplacian-of-Gaussian (LoG) operator:
Mikolajczyk 2002
G22
![Page 9: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/9.jpg)
Approximate LoG
n LoG is expensive, so we approximate it with
Difference-of-Gaussians (DoG):
IGkGD *
![Page 10: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/10.jpg)
DoG Efficiency
n The smoothed images need to be computed in
any case for feature description.
n We need only to subtract two images.
![Page 11: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/11.jpg)
DoB Filter (`Difference of Boxes')
Bay et al., ECCV 2006
n Even faster approximation is using box filters (by
integral image)
![Page 12: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/12.jpg)
Integral Image Computation-
code example
![Page 13: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/13.jpg)
Integral Image Usage
![Page 14: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/14.jpg)
Scale-Space Construction
n First construct scale-space:
increasing
First octave Second octave
IG *
IkG *
IkG *2
IG *2
IkG *2
IkG *2 2
![Page 15: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/15.jpg)
Difference-of-Gaussianss
n Now take differences:
![Page 16: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/16.jpg)
Scale-Space Extrema
n Choose all extrema within 3x3x3 neighborhood.
n Low cost – only several usually checked
D
kD
2kD
![Page 17: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/17.jpg)
Detector SIFT Overview
1. Find Scale-Space Extrema
2. Keypoint Localization & Filtering – Improve keypoints and throw out bad ones
3. Orientation Assignment
– Remove effects of rotation and scale
4. Create descriptor
– Using histograms of orientations
Descriptor
![Page 18: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/18.jpg)
Keypoint Localization & Filtering
n Now we have much less points than pixels.
n However, still lots of points (~1000s)…
– With only pixel-accuracy at best
• At higher scales, this corresponds to several pixels in base
image
– And this includes many bad points
Brown & Lowe 2002
![Page 19: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/19.jpg)
Keypoint Localization
n The problem:
x Sampling
Detected Extrema
True Extrema
![Page 20: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/20.jpg)
Keypoint Localization
n The solution:
– Take Taylor series expansion:
– Minimize to get true location of extrema:
xx
Dxx
x
DDxD
TT
T
2
2
2
1
Brown & Lowe 2002
x
D
x
Dx
1
2
2
ˆ
![Page 21: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/21.jpg)
Keypoints
(a) 233x189 image
(b) 832 DOG extrema
![Page 22: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/22.jpg)
Keypoint Filtering - Low Contrast
n Reject points with bad contrast
is smaller than 0.03 (image values in [0,1])
xD ˆ
![Page 23: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/23.jpg)
Keypoint Filtering - Edges
n Reject points with strong edge response in one
direction only
Point can move along edge
Point constrained
Point detection Point detection
![Page 24: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/24.jpg)
Keypoint Filtering
(c) 729 left after peak value threshold (from 832)
(d) 536 left after testing ratio of principle curvatures
![Page 25: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/25.jpg)
Detector SIFT Overview
1. Find Scale-Space Extrema
2. Keypoint Localization & Filtering
– Improve keypoints and throw out bad ones
3. Orientation Assignment
– Remove effects of rotation and scale
4. Create descriptor
– Using histograms of orientations
Descriptor
![Page 26: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/26.jpg)
Ideal Descriptors
n Robust to:
– Affine transformation
– Lighting
– Noise
n Distinctive
n Fast to match
– Not too large
– Usually L1 or L2 matching
![Page 27: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/27.jpg)
Detector SIFT Overview
1. Find Scale-Space Extrema
2. Keypoint Localization & Filtering
– Improve keypoints and throw out bad ones
3. Orientation Assignment – Remove effects of rotation and scale
4. Create descriptor
– Using histograms of orientations
Descriptor
![Page 28: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/28.jpg)
Orientation Assignment
n Now we have set of good points
n Choose a region around each point
– Remove effects of scale and rotation
![Page 29: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/29.jpg)
Orientation Assignment
n Use scale of point to choose correct image:
n Compute gradient magnitude and orientation
using finite differences:
yxIyxGyxL ,*,,,
yxLyxL
yxLyxLyx
yxLyxLyxLyxLyxm
,1,1
)1,(1,tan,
)1,(1,,1,1,
1
22
![Page 30: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/30.jpg)
Orientation Assignment
n Create gradient histogram (36 bins) – Weighted by magnitude and Gaussian window ( is 1.5 times
that of the scale of a keypoint)
![Page 31: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/31.jpg)
Orientation Assignment
n Any peak within 80% of the highest peak is used
to create a keypoint with that orientation
n ~15% assigned multiplied orientations, but
contribute significantly to the stability
n Finally a parabola is fit to the 3 histogram values
closest to each peak to interpolate the peak
position for better accuracy
![Page 32: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/32.jpg)
Detector SIFT Overview
1. Find Scale-Space Extrema
2. Keypoint Localization & Filtering
– Improve keypoints and throw out bad ones
3. Orientation Assignment
– Remove effects of rotation and scale
4. Create descriptor – Using histograms of orientations
Descriptor
![Page 33: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/33.jpg)
SIFT Descriptor
n Each point so far has x, y, σ, m, θ
n Now we need a descriptor for the region
– Could sample intensities around point, but…
• Sensitive to lighting changes
• Sensitive to slight errors in x, y, θ
n Look to biological vision
– Neurons respond to gradients at certain frequency and
orientation
• But location of gradient can shift slightly!
Edelman et al. 1997
![Page 34: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/34.jpg)
SIFT Descriptor
n 4x4 Gradient window
n Histogram of 4x4 samples per window in 8 directions
n Gaussian weighting around center( is 0.5 times that of the scale of
a keypoint)
n 4x4x8 = 128 dimensional feature vector
Image from: Jonas Hurrelmann
![Page 35: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/35.jpg)
SIFT Descriptor – Lighting changes
n Gains do not affect gradients
n Normalization to unit length removes contrast
n Saturation affects magnitudes much more than
orientation
n Threshold gradient magnitudes to 0.2 and renormalize
![Page 36: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/36.jpg)
Performance
n Very robust
– 80% Repeatability at:
• 10% image noise
• 45° viewing angle
• 1k-100k keypoints in database
n Best descriptor in [Mikolajczyk & Schmid 2005]’s
extensive survey
n 28000+ citations on Google Scholar
![Page 37: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/37.jpg)
Typical Usage
n For set of database images: 1. Compute SIFT features
2. Save descriptors to database
n For query image: 1. Compute SIFT features
2. For each descriptor: • Find a match
3. Verify matches • Geometry
• Hough transform
![Page 38: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/38.jpg)
Matching Descriptors
n Threshold on Distance – bad performance
n Nearest Neighbor – better
n Ratio Test – best performance
![Page 39: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/39.jpg)
Matching Descriptors - Distance
n L2 norm – used by Lowe
n SIFTDIST: linear time EMD algorithm that adds robustness to orientation shifts
Pele and Werman, ECCV 2008
![Page 40: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/40.jpg)
Ratio Test
n Need to be careful with the definition of next closest:
Best Match
False 2nd
best match
True 2nd
best match
Image 2 Image 1
![Page 41: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/41.jpg)
Fast Nearest-Neighbor Matching to
Feature Database
n Hypotheses are generated by approximate nearest neighbor
matching of each feature to vectors in the database
– SIFT use best-bin-first (Beis & Lowe, 97) modification to k-d
tree algorithm
– Use heap data structure to identify bins in order by their
distance from query point
n Result: Can give speedup by factor of 1000 while finding
nearest neighbor (of interest) 95% of the time
![Page 42: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/42.jpg)
3D Object Recognition
n Only 3 keys are needed for
recognition, so extra keys
provide robustness
![Page 43: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/43.jpg)
Recognition under occlusion
![Page 44: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/44.jpg)
Test of illumination Robustness
n Same image under differing illumination
273 keys verified in final match
![Page 45: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/45.jpg)
Location recognition
![Page 46: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/46.jpg)
Image Registration Results
[Brown & Lowe 2003]
![Page 47: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/47.jpg)
Cases where SIFT didn’t work
![Page 48: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/48.jpg)
n Same object under differing illumination
Cases where SIFT didn’t work
![Page 49: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/49.jpg)
Large illumination change
n Same object under differing illumination
n 43 keypoints in left image and the corresponding closest
keypoints on the right (1 for each)
![Page 50: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/50.jpg)
Large illumination change
n Same object under differing illumination
n 43 keypoints in left image and the corresponding closest
keypoints on the right (5 for each)
![Page 51: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/51.jpg)
Non rigid deformations
n 11 keypoints in left image and the corresponding closest
keypoints on the right (1 for each)
![Page 52: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/52.jpg)
Non rigid deformations
n 11 keypoints in left image and the corresponding closest
keypoints on the right (5 for each)
![Page 53: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/53.jpg)
Conclusion: SIFT
n Built on strong foundations
– First principles (LoG and DoG)
– Biological vision (Descriptor)
– Empirical results
n Many heuristic optimizations
– Rejection of bad points
– Sub-pixel level fitting
– Thresholds carefully chosen
![Page 54: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/54.jpg)
Conclusion: SIFT
n In wide use both in academia and industry
n Many available implementations:
– Binaries available at Lowe’s website
– C/C++ open source by A. Vedaldi (UCLA)
– C# library by S. Nowozin (Tu-BerlinMicrosoft)
n Protected by a patent
![Page 55: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/55.jpg)
Conclusion: SIFT
n Empirically found (Mikolajczyk & Schmid 2005) to show very
good performance, robust to image rotation, scale, intensity
change, and to moderate affine transformations
Scale = 2.5
Rotation = 450
![Page 56: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/56.jpg)
A note regarding invariance/robustness
n There is a tradeoff between invariance and
distinctiveness.
n For some tasks it is better not to be invariant
n Local features and kernels for classification of
texture and object categories: An in-depth
study - Zhang, Marszalek, Lazebnik and Schmid. IJCV 2007.
n 11 color names - J. van de Weijer, C. Schmid, Applying
Color Names to Image Description. ICIP 2007
![Page 57: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/57.jpg)
SIFT extensions
![Page 58: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/58.jpg)
Color
n Color SIFT - G. J. Burghouts and J. M. Geusebroek.
Performance evaluation of local colour invariants.
Comput. Vision Image Understanding, 2009
n Hue and Opponent histograms - J. van de Weijer,
C. Schmid. Coloring Local Feature Extraction.
ECCV 2006
n 11 color names - J. van de Weijer, C. Schmid,
Applying Color Names to Image Description. ICIP 2007
![Page 59: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/59.jpg)
PCA-SIFT
n Only change step 4 (creation of descriptor)
n Pre-compute an eigen-space for local gradient
patches of size 41x41
n 2x39x39=3042 elements
n Only keep 20 components
n A more compact descriptor
n In K.Mikolajczyk, C.Schmid 2005 PCA-SIFT
tested inferior to original SIFT
![Page 60: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/60.jpg)
Speed Improvements
n SURF - Bay et al. 2006
n Approx SIFT - Grabner et al. 2006
n GPU implementation - Sudipta N. Sinha et al. 2006
![Page 61: SIFT - The Scale Invariant Feature Transformmoodlearn.ariel.ac.il/pluginfile.php/829096/mod_resource/content/0/SIFT.pdfFast Nearest-Neighbor Matching to Feature Database n Hypotheses](https://reader033.vdocuments.net/reader033/viewer/2022042713/5fa85592a96a7e26790f39c5/html5/thumbnails/61.jpg)
GLOH (Gradient location-orientation
histogram)
17 location bins
16 orientation bins
Analyze the 17x16=272-d
eigen-space, keep 128 components
SIFT