lecture 03 local feature descriptors and … 03...lin zhang, sse, 2020 lecture 3 local feature...
TRANSCRIPT
Lin ZHANG, SSE, 2020
Lecture 3Local Feature Descriptors and Matching
Lin ZHANG, PhDSchool of Software Engineering
Tongji UniversitySpring 2020
Lin ZHANG, SSE, 2020
Content
• Scale Invariant Feature Transform• Case Study: Homography Estimation
• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
SIFT
• Scale Invariant Feature Transform• Proposed in [1]• It uses extrema of DoG to detect key points and the associated characteristic scales
• It uses SIFT to describe a key point
[1] D.G. Lowe, Distinctive image features from scale‐invariant keypoints, IJCV 60 (2), pp. 91‐110, 2004 (cited number: 55372 by Feb. 14, 2020)
Prof. David LoweUniversity of British Columbia
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
Assign Keypoints Orientations
• Assign orientation to the keypoint• Find local orientation: dominant orientation of gradient for
the image patch (its size is determined by the characteristic scale)
• Rotate the patch according to this angle; this can achieve rotation invariance description
Lin ZHANG, SSE, 2020
Assign Keypoints Orientations
• Orientation normalization• Compute orientation histogram• Select dominant orientation • Normalization: rotate the patch to the selected orientation
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
SIFT
• Building the descriptor• Sample the points around the keypoint• Rotate the gradients and coordinates by the previously computed orientation
• Separate the region in to sub‐regions• Create gradient‐orientation histogram for each sub‐region with 8 bins (In real implementation, each sample point is weighted by a Gaussian)
4 4
Lin ZHANG, SSE, 2020
SIFT
• Building the descriptor
• Actual implementation uses 4*4 sub regions which lead to a 4*4*8 = 128 element vector
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
SIFT
Lin ZHANG, SSE, 2020
Applications of SIFT
• Object recognition• Robot localization and mapping• Panorama stitching• 3D scene modeling, recognition and tracking• Analyzing the human brain in 3D magnetic resonance images
Lin ZHANG, SSE, 2020
Applications of SIFT
• Object recognition
Lin ZHANG, SSE, 2020
Applications of SIFT
• Object recognition
Lin ZHANG, SSE, 2020
Content
• Scale Invariant Feature Transform• Case Study: Homography Estimation
• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Matrix differentiation
• Function is a vector and the variable is a scalar
1 2( ) ( ), ( ),..., ( ) Tnf t f t f t f t
Definition
1 2 ( )( ) ( ), ,...,T
ndf tdf t df tdfdt dt dt dt
Lin ZHANG, SSE, 2020
• Function is a matrix and the variable is a scalar11 12 1
21 22 2
1 2
( ) ( ),..., ( )( ) ( ),..., ( )
( ) ( )
( ) ( ),..., ( )
m
mij n m
n n nm
f t f t f tf t f t f t
f t f t
f t f t f t
Definition111 12
221 22
1 2
( )( ) ( ) ,...,
( )( ) ( ) ( ),...,
( ) ( ) ( ),...,
m
mij
n m
n n nm
df tdf t df tdt dt dt
df tdf t df t df tdfdt dt dt
dt dt
df t df t df tdt dt dt
Matrix differentiation
Lin ZHANG, SSE, 2020
• Function is a scalar and the variable is a vector
1 2( ), ( , ,..., )Tnf x x xx x
Definition
1 2
, ,...,T
n
df f f fd x x x
x
In a similar way,
1 2( ), ( , ,..., )nf x x xx x
1 2
, ,...,n
df f f fd x x x
x
Matrix differentiation
Lin ZHANG, SSE, 2020
• Function is a vector and the variable is a vector 1 2 1 2, ,..., , ( ), ( ),..., ( )T T
n mx x x y y y x y x x xDefinition
1 1 1
1 2
2 2 2
1 2
1 2
( ) ( ) ( ), ,...,
( ) ( ) ( ), ,...,
( ) ( ) ( ), ,...,
n
nT
m m m
n m n
y y yx x x
y y yd x x xd
y y yx x x
x x x
x x xyx
x x x
Matrix differentiation
Lin ZHANG, SSE, 2020
• Function is a vector and the variable is a vector 1 2 1 2, ,..., , ( ), ( ),..., ( )T T
n mx x x y y y x y x x xIn a similar way,
1 2
1 1 1
1 2
2 2 2
1 2
( ) ( ) ( ), ,...,
( ) ( ) ( ), ,...,
( ) ( ) ( ), ,...,
m
mT
m
n n n n m
y y yx x x
y y yd x x xd
y y yx x x
x x x
x x xyx
x x x
Matrix differentiation
Lin ZHANG, SSE, 2020
• Function is a vector and the variable is a vectorExample:
11 2 2
2 1 1 2 2 3 22
3
( ), , ( ) , ( ) 3
( )
xy
x y x x y x xy
x
xy x x x
x
1 2
1 11
1 2
2 23
1 2
3 3
( ) ( )
2 0( ) ( ) 1 3
0 2( ) ( )
T
y yx x x
d y yd x x
xy y
x x
x x
y x xx
x x
Matrix differentiation
Lin ZHANG, SSE, 2020
• Function is a scalar and the variable is a matrix
11 12 1
1 2
( ) n
m m mn
f f fx x x
dfd
f f fx x x
XX
( ), m nf X X
Definition
Matrix differentiation
Lin ZHANG, SSE, 2020
• Useful results1, nx a
,T Td d
d d
a x x aa ax x
Then,
How to prove?
(1)
Matrix differentiation
Lin ZHANG, SSE, 2020
• Useful results1,m n nA x (2) Then, T
dA Ad
xx
1,m n nA x (3) Then,T T
Td A Ad
xx
1,n n nA x (4) Then, ( )T
Td A A Ad
x x xx
1 1, ,m n m n X a b (5) Then,T
Tdd
a Xb abX
1 1, ,n m m n X a b (6) Then,T T
Tdd
a X b baX
1nx (7) Then, 2Td
d
x x xx
Matrix differentiation
Lin ZHANG, SSE, 2020
Content
• Scale Invariant Feature Transform• Case Study: Homography Estimation
• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Lagrange multiplier
• Single‐variable function
f(x) is differentiable in (a, b). At , f(x) achieves an extremum
0 ( , )x a b
0| 0x
dfdx
• Two‐variables function
f(x, y) is differentiable in its domain. At , f(x, y)achieves an extremum
0 0( , )x y
0 0 0 0( , ) ( , )| 0, | 0x y x yf fx y
Lin ZHANG, SSE, 2020
Lagrange multiplier
• In general case
( )f xIf , achieves a local extremum at x0 and it is derivable at x0, then x0 is a stationary point of f(x), i.g.,
0 0 01 2
| 0, | 0,..., | 0n
f f fx x x
x x x
Or in other words,
0( ) |f x xx 0
1nx
Lin ZHANG, SSE, 2020
Lagrange multiplier
• Lagrange multiplier is a strategy for finding the stationary point of a function subject to equality constraints
Problem: find stationary points for 1( ), ny f x x under m constraints ( ) 0, 1, 2,...,kg k m x
is a stationary point of with constraints
Solution:
11
( ; ,..., ) ( ) ( )m
m k kk
F f g
x x xIf is a stationary point of F, then,
0 10 20 0( , , ..., )m x
0x ( )f xJoseph‐Louis LagrangeJan. 25, 1736~Apr.10, 1813
Lin ZHANG, SSE, 2020
Lagrange multiplier
• Lagrange multiplier is a strategy for finding the stationary point of a function subject to equality constraints
Solution:1
1( ; ,..., ) ( ) ( )
m
m k kk
F f g
x x x
is a stationary point of F0 10 0( , ,..., )m x
1 2 1 2
0, 0,..., 0, 0, 0,..., 0n m
F F F F F Fx x x
n + m equations!at that point
Problem: find stationary points for 1( ), ny f x x under m constraints ( ) 0, 1, 2,...,kg k m x
Lin ZHANG, SSE, 2020
Lagrange multiplier
• ExampleProblem: for a given point p0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p0.
y=xp0
?
The distance is 2 2( , ) ( 1) ( 0)f x y x y
Now we want to find the stationary point of f(x, y) under the constraint
( , ) 0g x y y x According to Lagrange multiplier method, construct another function
2 2( , , ) ( ) ( ) ( 1) ( )F x y f x g x x y y x Find the stationary point for ( , , )F x y
Lin ZHANG, SSE, 2020
Lagrange multiplier
• ExampleProblem: for a given point p0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p0.
y=xp0
?
0
0
0
FxFyF
2( 1) 02 0
0
xy
x y
0.50.51
xy
(0.5,0.5,1) is a stationary point of ( , , )F x y (0.5,0.5) is a stationary point of f(x,y) under constraints
Lin ZHANG, SSE, 2020
Content
• Scale Invariant Feature Transform• Case Study: Homography Estimation
• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
LS for Inhomogeneous Linear System
Consider the following linear equations system
1 2 1
1 2 2
3 1 1 32 4 2 1 4x x x
x x x
Matrix form: A x bA x b
It can be easily solved 1
2
12
xx
Lin ZHANG, SSE, 2020
LS for Inhomogeneous Linear System
How about the following one?
1 21
1 22
1 2
3 1 1 32 4 2 1 4
1 2 62 6
x xx
x xx
x x
It does not have a solution!
What is the condition for a linear equation system can be solved?
A x b
Can we solve it in an approximate way?A: we can use least squares technique!
Carl Friedrich Gauss
Lin ZHANG, SSE, 2020
LS for Inhomogeneous Linear System
Let’s consider a system of p linear equations with q unknowns
11 1 12 2 1 1
21 1 22 2 2 2
1 1 2 2
...
...
......
q q
q q
p p pq q p
a x a x a x
a x a x a xA
a x a x a x
b
bx b
b
We consider the case: p>q, and rank(A)=qIn general case, there is no solution!
Instead, we want to find a vector x that minimizes the error:22
1 11
( ) ( ... )p
i iq q ii
E a x a x A
x b x b
unknowns
Lin ZHANG, SSE, 2020
LS for Inhomogeneous Linear System
2*2
arg min ( ) arg minE A x x
x x x b
1* T TA A A
x bPseudoinverse of A
How about the pseudoinverse of A when A is square and non-singular?
Lin ZHANG, SSE, 2020
LS for Homogeneous Linear System
Let’s consider a system of p linear equations with q unknowns
11 1 12 2 1
21 1 22 2 2
1 1 2 2
... 0
... 0
...... 0
q q
q q
p p pq q
a x a x a x
a x a x a xA
a x a x a x
x 0
We consider the case: p>q, and rank(A)=q
unknowns
Theoretically, there is only a trivial solution: x = 0
So, we add a constraint to avoid the trivial solution 2
1x
Lin ZHANG, SSE, 2020
LS for Homogeneous Linear System
We want to minimize , subject to 21x 2
2( )x xE A
*2
arg min ( ), . ., 1x
x x xE s t
Use the Lagrange multiplier to solve it,
2 2*2 2
arg min 1x
x x xA
(1)
Solving the stationary point of the Lagrange function,
2 2
2 2
2 2
2 2
1
10
x x0
x
x x
A
A
(3)
(2)
Lin ZHANG, SSE, 2020
LS for Homogeneous Linear System
Then, we have
2 2
2 21x x
0x
A
(3)
x xTA A
x is the eigen‐vector of ATA associated with the eigenvalue
2
2x x x x x xT T TE A A A
The unit vector x is the eigenvector associated with the minimum eigenvalue of TA A
Lin ZHANG, SSE, 2020
Content
• Scale Invariant Feature Transform• Case Study: Homography Estimation
• Matrix Differentiation• Lagrange Multiplier• Least‐squares for Linear Systems• RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
RANSAC‐based Homography Estimation
Problem definition: On two projective planes P1 and P2, there is a set of corresponding points , and we suppose that there is a homography matrix linking the two planes,
'1
,n
i i ix x
' , 1, 2,...,i iH i n x xCoordinates of and are known, ix 'ix we need to find H
11 12 13
21 22 23
31 32 33
a a aH a a a
a a a
Note: H is defined up to a scale factor. In other words, it has 8 DOFs
Lin ZHANG, SSE, 2020
RANSAC‐based Homography Estimation
Note: Theoretically speaking, homography can only be estimatedbetween two planes, i.e., when you use such a technique to stitch twoimages, image contents should be roughly on the same plane
Lin ZHANG, SSE, 2020
11 12 13
21 22 23
31 32 33 1
a a acu xcv a a a yc a a a
11 12 13
21 22 23
31 32 33
a x a y a cua x a y a cva x a y a c
11 12 13
31 32 33
21 22 23
31 32 33
a x a y a ua x a y aa x a y a va x a y a
4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom 11
12
13
21
22
23
31
32
33
1 0 0 00
0 0 0 1
aaaa
x y ux uy ua
x y vx vy vaaaa
Thus, four correspondence pairs generate 8 equations
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom
0 (1)A x
8 9 9 1
Normally, ; thus (1) has 1 (9‐8) solution vector in its solution space
( ) 8Rank A
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
• How about the case when there are more than 4 correspondence pairs?• Use the LS method to solve the model directly?• NO! Because usually, outliers exist among the correspondence pairs
RANdom SAmple Consensus (RANSAC) framework is a good candidate to solve this kind of issues
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
ObjectiveRobust fit a model to a data set S which contains outliersAlgorithm(1) Randomly select a sample of s data points from S and instantiate the
model from this subset(2) Determine the set of data points Si which are within a distance
threshold t of the model. The set Si is the consensus set of the sample and defines the inliers of S
(3) If the size of Si (the number of inliers) is greater than some threshold T, re‐estimate the model using all points in Si and terminate
(4) If the size of Si is less than T, select a new subset and repeat the above
(5) After N trials the largest consensus set Si is selected, and the model is re‐estimated using all points in the subset Si
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Line fitting: least square
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Line fitting: RANSAC
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Line fitting by RANSAC
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Line fitting by RANSAC
• Randomly select two points
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Line fitting by RANSAC
• Randomly select two points• The hypothesized model is the
line passing through the two points
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Line fitting by RANSAC
• Randomly select two points• The hypothesized model is the
line passing through the two points
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Line fitting by RANSAC
• Randomly select two points• The hypothesized model is the
line passing through the two points
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Line fitting by RANSAC
• Test another two points
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Line fitting by RANSAC
• The final fitting result
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
Line fitting by RANSAC
• The final fitting result
RANSAC‐based Homography Estimation
Lin ZHANG, SSE, 2020
RANSAC‐based Homography Estimation
Can you describe the steps of homography estimation when
using RANSAC?
Lin ZHANG, SSE, 2020
Homography Estimation: Example 1
Lin ZHANG, SSE, 2020
Interest points detection
Homography Estimation: Example 1
Lin ZHANG, SSE, 2020
Correspondence estimation
Then, the homography matrix can be estimated by using the correspondence pairs with RANSAC
Homography Estimation: Example 1
Lin ZHANG, SSE, 2020
Transform image one using the estimated homography matrix
Homography Estimation: Example 1
Lin ZHANG, SSE, 2020
Finally, stitch the transformed image one with image two
Homography Estimation: Example 1
Lin ZHANG, SSE, 2020
Homography Estimation: Example 2
Lin ZHANG, SSE, 2020
Homography Estimation: Example 2Interest points detection
Lin ZHANG, SSE, 2020
Correspondence estimation
Then, the homography matrix can be estimated by using the correspondence pairs with RANSAC
Homography Estimation: Example 2
Lin ZHANG, SSE, 2020
Transform image one using the estimated homography matrix
Homography Estimation: Example 2
Lin ZHANG, SSE, 2020
Finally, stitch the transformed image one with image two
Homography Estimation: Example 2
Lin ZHANG, SSE, 2020
Homography Estimation: Example 3Project products of students from 2009 Media&Arts