lecture 4.3 estimating homographies from feature ... - uio
TRANSCRIPT
Lecture 4.3 Estimating homographies from feature correspondences Thomas Opsahl
Homographies induced by central projection
2
𝐻 𝐻
• Homography 𝐻𝒖� = 𝒖�′
𝐻 =ℎ1 ℎ2 ℎ3ℎ4 ℎ5 ℎ6ℎ7 ℎ8 ℎ9
• Point-correspondences can be determined automatically • Erroneous correspondences are common • Robust estimation is required to find 𝐻
1 1 2 2
2 3 1
𝒖 𝒖′
Homographies induced by central projection
3
𝐻
1 2 3
𝐻1←3 𝐻1←2
𝐻 1 2 1 2
• Homography 𝐻𝒖� = 𝒖�′
𝐻 =ℎ1 ℎ2 ℎ3ℎ4 ℎ5 ℎ6ℎ7 ℎ8 ℎ9
• Point-correspondences can be determined automatically • Erroneous correspondences are common • Robust estimation is required to find 𝐻
𝒖 𝒖′
Estimating the homography between overlapping images
• Establish point correspondences 𝒖𝑖 ↔ 𝒖′𝑖 – Find key points 𝒖𝑖 ∈ Img1 and 𝒖′𝑖 ∈ Img2 – Represent key points by suitable descriptors – Determine correspondences 𝒖𝑖 ↔ 𝒖′𝑖 by matching descriptors – Some wrong correspondences are to be expected
• Estimate the homography 𝐻 such that 𝒖′𝑖 = 𝐻𝒖𝑖 ∀𝑖
– Robust estimation with RANSAC – Improved estimation based on RANSAC inliers
• This homography enables us to compose the images into a
larger image – Image mosaicing – Panorama
4
Adaptive RANSAC
Objective To robustly fit a model 𝒚 = 𝑓 𝒙;𝜶 to a data set 𝑆 containing outliers Algorithm 1. Let 𝑁 = ∞, 𝑆𝐼𝐼 = ∅ and #𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 = 0
2. while 𝑁 > #𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 repeat 3-5
3. Estimate parameters 𝜶𝑡𝑡𝑡 from a random 𝑖-tuple from 𝑆
4. Determine inlier set 𝑆𝑡𝑡𝑡, i.e. data points within a distance 𝑖 of the model 𝒚 = 𝑓 𝒙;𝜶𝑡𝑡𝑡
5. If 𝑆𝑡𝑡𝑡 > 𝑆𝐼𝐼 , set 𝑆𝐼𝐼 = 𝑆𝑡𝑡𝑡, 𝜶 = 𝜶𝑡𝑡𝑡, 𝜔 = 𝑆𝐼𝐼
𝑆 and 𝑁 = 𝑙𝑙𝑙 1−𝑝
𝑙𝑙𝑙 1−𝜔𝑛 with 𝑝 = 0.99 Increase #𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖𝑖 by 1
5
Estimating the homography
• Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for 𝑖 point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography
6
Estimating the homography
• Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for 𝒏 point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography
• The homography has 8 degrees of freedom, but it is custom to treat all 9 entries of the matrix
as unknowns instead of setting one of the entries to 1 which excludes all potential solutions where this entry is 0
• Let us solve the equation 𝐻𝒖� = 𝒖�′ for the entries of the homography matrix 𝐻𝒖� = 𝒖�′ ℎ1 ℎ2 ℎ3ℎ4 ℎ5 ℎ6ℎ7 ℎ8 ℎ9
𝑢𝑣1
=𝑢′𝑣′1
7
Basic homography estimation
8
1
2
3
1 2 3 1 2 3 4
4 5 6 4 5 6 5
7 8 9 7 8 9 6
7
8
9
0 0 0 11 0 0 0
1 1 1 0 0 0
hhh
h h h u u uh vh h u u v v u v v v hh h h v v uh vh h v u v u u u v u hh h h uh vh h v u v v v u u u v u h
hhh
′ ′ ′ ′ ′+ + = − − − ′ ′ ′ ′ ′= ⇔ + + = ⇔ − − − ′ ′ ′ ′ ′ ′ + + = − − −
000
A
= ⇔ =
h 0
Basic homography estimation
Observe that the third row in 𝐴 is a linear combination of the first and second row 𝑖𝑖𝑟3 = −𝑢′ ∙ 𝑖𝑖𝑟1 − 𝑣′ ∙ 𝑖𝑖𝑟2
Hence every correspondence 𝒖𝑖 ↔ 𝒖′𝑖 contribute with 2 equations in the 9 unknown entries
9
1
2
3
1 2 3 1 2 3 4
4 5 6 4 5 6 5
7 8 9 7 8 9 6
7
8
9
0 0 0 11 0 0 0
1 1 1 0 0 0
hhh
h h h u u uh vh h u u v v u v v v hh h h v v uh vh h v u v u u u v u hh h h uh vh h v u v v v u u u v u h
hhh
′ ′ ′ ′ ′+ + = − − − ′ ′ ′ ′ ′= ⇔ + + = ⇔ − − − ′ ′ ′ ′ ′ ′ + + = − − −
000
A
= ⇔ =
h 0
Basic homography estimation
• Since 𝐻 (and thus 𝒉) is homogeneous, we only need the matrix 𝐴 to have rank 8 in order to determine 𝒉 up to scale
• It is sufficient with 4 point correspondences where no 3 points are collinear
• We can calculate the non-trivial solution to the equation 𝐴𝒉 = 𝟎 by SVD
svd 𝐴 = 𝑈𝑆𝑉𝑇
• The solution is given by the right singular vector without a singular value which is the last column of 𝑉, i.e. 𝒉 = 𝒗9
10
1
2
31 1 1 1 1 1 1
41 1 1 1 1 1 1
52 2 2 2 2 2 2
62 2 2 2 2 2 2
7
8
9
0 0 0 1 01 0 0 0 0
0 0 0 1 01 0 0 0 0
hhhu v v u v v vhu v u u u v uhu v v u v v vhu v u u u v uhhhA
′ ′ ′ − − − ′ ′ ′− − − ′ ′ ′ =− − − ′ ′ ′− − −
=
h 0
Basic homography estimation
• Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for 𝒏 point-correspondences 2. A way to determine which of the point correspondences that are inliers for a given homography
Direct Linear Transform
1. Build the matrix 𝐴 from at least 4 point-correspondences 𝑢𝑖 ,𝑣𝑖 ↔ 𝑢′𝑖 ,𝑣′𝑖 2. Obtain the SVD of 𝐴: 𝐴 = 𝑈𝑆𝑉𝑇 3. If 𝑆 is diagonal with positive values in descending order along the main diagonal,
then 𝒉 equals the last column of 𝑉 4. Reconstruct 𝐻 from 𝒉
1 1 1 1 1 1 1
1 1 1 1 1 1 1
0 0 0 11 0 0 0
u v v u v v vA u v u u u v u
′ ′ ′− − − ′ ′ ′= − − −
11
Basic homography estimation
• The basic DLT algorithm is never used with more than 4 point-correspondences • This is because the algorithm performs better when all the terms of 𝐴 has a similar scale
– Note that some of the terms will always be of scale 1 • To achieve this, it is common to extend the algorithm with a normalization and a
denormalization step
Normalized Direct Linear Transform 1. Normalize the set of points 𝒖𝑖 = 𝑢𝑖 , 𝑣𝑖 𝑇 by computing a similarity transform 𝑇 that
translates the centroid to the origin and scales such that the average distance from the origin is 2
2. In the same way normalize the set of points 𝒖′𝑖 = 𝑢′𝑖 ,𝑣′𝑖 𝑇 by computing a similarity transform 𝑇′
3. Apply the basic DLT algorithm on the normalized points to obtain a homography 𝐻� 4. Denormalize to get the homography: 𝐻 = 𝑇′−1𝐻�𝑇
12
Basic homography estimation
• Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for 𝑖 point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography
• For a point correspondence 𝑢𝑖 , 𝑣𝑖 ↔ 𝑢′𝑖 , 𝑣′𝑖 and homography 𝐻, we can choose from
several errors – Algebraic error: 𝜀𝑖 = 𝐴𝑖𝒉 where
– Geometric errors: 1. 𝜀𝑖 = 𝑑 𝐻𝒖𝑖 ,𝒖′𝑖 + 𝑑 𝒖𝑖 ,𝐻−1𝒖′𝑖 (Reprojection error) 2. 𝜀𝑖 = 𝑑 𝒖𝑖 ,𝐻−1𝒖′𝑖 3. 𝜀𝑖 = 𝑑 𝐻𝒖𝑖 ,𝒖′𝑖
0 0 0 11 0 0 0
i i i i i i ii
i i i i i i i
u v v u v v vA
u v u u u v u′ ′ ′− − −
= ′ ′ ′− − −
13
Notation
Euclidean distance 𝑑 ∙,∙
Inhomogenous 𝐻𝒖�𝑖 𝐻𝒖𝑖
Inhomogeneous 𝐻−1𝒖�′𝑖 𝐻−1𝒖′𝑖
Robust homography estimation
• Finally we would typically re-estimate 𝐻 from all correspondences in 𝑆𝐼𝐼 – Normalized DLT – Minimize ϵ = ∑𝜖𝑖 in an iterative optimization method like Levenberg Marquardt
RANSAC estimation of homography For a set of point-correspondences S = 𝒖𝒊 ↦ 𝒖′𝑖 , perform 𝑁 iterations, where 𝑁 is determined adaptively
1. Estimate 𝐻𝑡𝑡𝑡 from 4 random correspondences 𝒖𝒊 ↦ 𝒖′𝑖 using the basic DLT algorithm 2. Determine the set of inlier-correspondences 𝑆𝑡𝑡𝑡 = 𝒖𝒊 ↦ 𝒖′𝑖 𝑖𝑢𝑠ℎ 𝑖ℎ𝑖𝑖 𝜖𝑖 < 𝑖
Here one can choose 𝜖𝑖 = 𝑑 𝐻𝒖𝑖 ,𝒖′𝑖 + 𝑑 𝒖𝑖 ,𝐻−1𝒖′𝑖 and 𝑖 = 5.99𝜎 where 𝜎 is the expected uncertainty in key-point positions
3. If 𝑆𝑡𝑡𝑡 > 𝑆𝐼𝐼 update 𝑁, homography and inlier set: 𝐻 = 𝐻𝑡𝑡𝑡, 𝑆𝐼𝐼 = 𝑆𝑡𝑡𝑡
14
Image mosaicing
15
• Let us compose these two images into a larger image
Image mosaicing
• Find key points and represent by descriptors
16
Image mosaicing
• Establish point-correspondences by matching descriptors • Several wrong correspondences
17
Image mosaicing
• Establish point-correspondences by matching descriptors • Several wrong correspondences
18
Image mosaicing
• Estimate homography 𝐻𝒖� = 𝒖�′ – OpenCV
#include "opencv2/calib3d.hpp" cv::findHomography(srcPoints, dstPoints, CV_RANSAC);
– Matlab tform = estimateGeometricTransform(srcPoints,dstPoints,’projective’);
19
𝐻
Image mosaicing
• Represent the images in common coordinates (Note the additional translation!) – OpenCV
#include "opencv2/calib3d.hpp" cv::warpPerspective(img1, img2, H, output_size);
– Matlab img2 = imwarp(img1,tform);
20
Image mosaicing
• Now we can compose the images
21
22
Overwriting
23
Blending with a ramp
24
Blending with a ramp + histogram equalization
SVD How to use • Matlab
[U,S,V] = svd(A); Right singular vectors are columns in V
• OpenCV cv::SVD::compute(A, S, U, Vtranspose, cv::SVD::FULL_UV); Right singular vectors are rows in Vtranspose
• Eigen Eigen::JacobiSVD<Eigen::MatrixXd> svd(A, Eigen::ComputeFullU | Eigen::ComputeFullV); Right singular vectors are columns in svd.matrixV()
25
Singular Value Decomposition The singular value decomposition of a real 𝑚 × 𝑖 matrix 𝐴 is a factorization 𝐴 = 𝑈S𝑉𝑇 Here 𝑈 is a orthogonal 𝑚 × 𝑚 matrix, 𝑉 is a orthogonal 𝑖 × 𝑖 matrix and 𝑆 is a real positive diagonal 𝑚 × 𝑖 matrix The diagonal entries of 𝑆 = diag 𝑖1, … , 𝑖min 𝑚,𝑛 are known as the singular values of 𝐴 and the columns of 𝑈 = 𝒖1, … ,𝒖𝑚 and 𝑉 = 𝒗1, … ,𝒗𝑛 are known as the left and right singular vectors of 𝐴 respectively The nullspace of A is the span of the right singular vectors 𝒗𝑖 that corresponds to a zero singular value 𝑖𝑖 (or does not have a corresponding singular value)
SVD Singular Value Decomposition The singular value decomposition of a real 𝑚 × 𝑖 matrix 𝐴 is a factorization 𝐴 = 𝑈S𝑉𝑇 Here 𝑈 is a orthogonal 𝑚 × 𝑚 matrix, 𝑉 is a orthogonal 𝑖 × 𝑖 matrix and 𝑆 is a real positive diagonal 𝑚 × 𝑖 matrix The diagonal entries of 𝑆 = diag 𝑖1, … , 𝑖min 𝑚,𝑛 are known as the singular values of 𝐴 and the columns of 𝑈 = 𝒖1, … ,𝒖𝑚 and 𝑉 = 𝒗1, … ,𝒗𝑛 are known as the left and right singular vectors of 𝐴 respectively The nullspace of A is the span of the right singular vectors 𝒗𝑖 that corresponds to a zero singular value 𝑖𝑖 (or does not have a corresponding singular value)
Applications of SVD Solving homogeneous linear equations like
𝐴𝒉 = 𝟎 Method For theoretical problems, 𝐡 ∈ null 𝐴 so 𝒉 is a linear combination of the right singular vectors 𝒗𝑖 that correspond to a zero singular value 𝑖𝑖
𝒉 = �𝑘𝑖𝒗𝑖 ; 𝑘𝑖 ∈ ℝ, 𝑖𝑖 = 0 (or missing)
For practical problems, the presence of noise force us to expand the solution by including those right singular vectors that correspond to small singular values 𝑖𝑖 ≈ 0
𝒉 = �𝑘𝑖𝒗𝑖 ; 𝑘𝑖 ∈ ℝ, 𝑖𝑖 ≈ 0 (or missing)
26
SVD Example
From [U,S,V] = svd(A); we get
From this we see that A has: • 2 left singular vectors
• 2 nonzero singular values • 3 right singular vectors Since 𝒗3 does not have a corresponding singular value, 𝒙 = 𝒗3 is a non-trivial solution to A𝒙 = 0 and 𝒙 = 𝑘 ∙ 𝒗3; 𝑘 ∈ ℝ\ 0 is the family of all non-trivial solutions
27
1 2 3 04 5 6 0
Axyz
=
=
x 0
0.3863 0.9224 9.5080 0 00.9224 0.3863 0 0.7729 0
0.4287 0.8060 0.40820.5663 0.1124 0.81650.7039 0.5812 0.4082
U S
V
− − = = −
− = − − − −
1 2 3
0.4287 0.8060 0.40820.5663 0.1124 0.81650.7039 0.5812 0.4082
− = − = = − − −
v v v
1 29.5080 0.7729s s= =
1 2
0.3863 0.92240.9224 0.3863− −
= = − u u
SVD Example
This time singular value decomposition give us the following singular values and right singular vectors:
This time all right singular vectors correspond to a non-zero singular value, so the equation does not have any non-trivial solutions!
If this equation came from a practical problem, instead of looking for solutions to 𝐴𝒙 = 0, we might be looking for the 𝒙 that minimize 𝐴𝒙 Since 𝑖1 ≉ 0, 𝑖2 ≉ 0, 𝑖3 ≈ 0, we would conclude that 𝒙 = 𝒗3 solves the equation in a least-squares sense Check:
28
1.0792 2.0656 3.08494.0959 5.0036 6.0934 01.0679 2.0743 3.0655 04.0758 5.0392 6.0171
A
xyz
=
=
x 0
1 2 3
1 2 3
13.6295 1.0849 0.05060.4336 0.8103 0.39420.5635 0.0975 0.82030.7032 0.5778 0.4143
s s s= = =
− − = − = − = − −
v v v 3
0.00910.42880.01050.0341
A
= − −
v
Summary
• Homography 𝐻𝒖� = 𝒖�′
𝐻 =ℎ1 ℎ2 ℎ3ℎ4 ℎ5 ℎ6ℎ7 ℎ8 ℎ9
• Automatic point-correspondences
• Wrong correspondences are common
• RANSAC estimation
– Basic DLT (Direct Linear Transform) on 4 random correspondences
– Inliers determined from the reprojection error 𝜖𝑖 = 𝑑 𝐻𝒖𝑖 ,𝒖′𝑖 + 𝑑 𝒖𝑖 ,𝐻−1𝒖′𝑖
• Improve estimate by normalized DLT on inliers or iterative methods for an even better estimate
• Additional reading – Szeliski: 6.1.1 – 6.1.3
29