ecen631 15 - 3-d reconstructionece631web.groups.et.byu.net/lectures/ecen631 15 - 3-d... · 2012. 2....
TRANSCRIPT
![Page 1: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/1.jpg)
3D Reconstruction
![Page 2: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/2.jpg)
3-D Reconstruction Depending on the applica/on, disparity map may be sufficient for interpre/ng 3D informa/on.
Most research work focus on solving the correspondence problem.
For robo/c vision applica/ons such as 3D object matching, 3D modeling, robot grasping and naviga/on, actual 3D reconstruc/on or depth map is needed.
2/27/12 2 ECEn 631
![Page 3: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/3.jpg)
2/27/12 ECEn 631 3
![Page 4: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/4.jpg)
2/27/12 ECEn 631 4
![Page 5: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/5.jpg)
3D Reconstruction Correspondence Problem Epipolar Geometry 3D Reconstruc9on
Both intrinsic and extrinsic parameters are known – Easy to calibrate for stereo vision (fixed cameras) – Unambiguous (Eculidean structure)
Only the intrinsic parameters are known and es9mate extrinsic parameters (camera mo9on is not known) – Structure from mo9on using a single camera – Up to an unknown scaling factor (affine structure)
Only the pixel correspondences are the only informa9on available (video from unknown sources) – Up to an unknown projec9ve transforma9on of the environment
2/27/12 ECEn 631 5
![Page 6: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/6.jpg)
3-D Reconstruction
Euclidean Structure Affine Structure (up to a scale factor) Projec/ve Transforma/on
2/27/12 6 ECEn 631
![Page 7: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/7.jpg)
Ambiguities and Constraints
If M=BC, then for any inver/ble matrix H.
One cannot dis/nguish the pair from from measurements of M.
However, if the factors are constrained to have a par/cular structure, then recovering the true unknown B and C may be possible.
For example, if we require that B be upper triangular and C be a rota/on matrix.
!
M = BC = (BH"1
)(HC) # ˜ B ˜ C
!
(B,C)
!
( ˜ B , ˜ C )
2/27/12 7 ECEn 631
![Page 8: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/8.jpg)
Projection Matrix
!
"
xi
yi
1
#
$
% % %
&
'
( ( (
=
fsx fs) ox
0 fsy oy
0 0 1
#
$
% % %
&
'
( ( (
1 0 0 0
0 1 0 0
0 0 1 0
#
$
% % %
&
'
( ( (
Rcw Tcw
0 1
#
$ %
&
' (
Xw
Yw
Zw
1
#
$
% % % %
&
'
( ( ( (
!
"p = K#ogPw = MPw
Defines the relation between image pixel and 3-D world
Unknown scale factor
Intrinsic Parameter Matrix
Projection Matrix
Extrinsic Parameter Matrix
2/27/12 8 ECEn 631
![Page 9: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/9.jpg)
The ambigui/es associated with R0 and gw are rather harmless.
With the presence of ambiguity H, it is possible to obtain “some” 3‐D reconstruc9on, one that differs from the true Euclidean reconstruc9on by a geometric transforma9on.
The richer the class of transforma9on, the easier the reconstruc9on, the larger the ambiguity.
!
"p = K#ogPw
= KR0
$1R0#oH
$1Hggw
$1gwPw
2/27/12 9 ECEn 631
![Page 10: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/10.jpg)
Geometric Stratification
!
"i p i = Ki#ogiePe , i = 1, 2For two views, i=2. Changed from w to e to indicate Euclidean.
!
Mie
Choose camera 1 to be the world frame, i.e., .
!
g1e = (T ,R) = (I,0)
!
"1p 1
= K1#oPe = M
1ePe $ M1e = K1#o = K
1I,0[ ]
Ambiguity still exists.
!
K1"oPe = K
1"oH
#1HPe = M
1pPp
!
M1p
!
Pp
!
K2"og2ePe = K
2"og2eH
#1HPe = M
2 pPp
!
M2 p
!
Pp
Changed from e to p to indicate Projective transformation.
2/27/12 10 ECEn 631
![Page 11: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/11.jpg)
Geometric Stratification
!
H"1
=K1
"10
vT
v4
#
$ %
&
' ( ) R
4*4Select H-1 to be
!
"2p 2
= K2#og
2ePe = K2#og
2eHa
$1H p
$1H pHagePw
!
Pp
!
Pe
!
"1p 1
= K1#oPe = K
1#oH
$1HPe = K
1#o
K1
$10
vT
v4
%
& '
(
) * Pp = #oPp
!
Pp
!
H"1
=K1
"10
0 1
#
$ %
&
' ( I 0
vT
v4
#
$ %
&
' ( = Ha
"1Hp
"1
!
Pa
2/27/12 11 ECEn 631
![Page 12: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/12.jpg)
Projection Matrices
!
Pp = HpHagePw
!
Pe!
Pa
Euclidean (calibrated) camera:
Affine (weakly calibrated) camera:
Projective (uncalibrated) camera:
!
Mip = Ki"ogieHa
#1Hp
#1
!
Mie = Ki"ogie
!
Mia = Ki"ogieHa
#1
2/27/12 12 ECEn 631
![Page 13: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/13.jpg)
Full Euclidean Reconstruction
2/27/12 13 ECEn 631
Canonical Configura/on ‐ Reprojec/on – Need a fully calibrated stereo system – Rec/fy both leT and right images – Search corresponding points along the horizontal lines – Compute a disparity map of the image or dispari/es of points of interest
– Reconstruct 3D General Configura/on ‐ Triangula/on
– Need a fully calibrated stereo system – No rec/fica/on is necessary (maintain image quality) – Search corresponding points along the epipolar lines – Reconstruct 3D using triangula/on
![Page 14: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/14.jpg)
2/27/12 ECEn 631 14
Camera Intrinsic and Distortion Parameters
Camera Intrinsic and Distortion Parameters
Stereo Calibration – R & T
Stereo Rectification – R1, R2, P1, P2, Q
Use R1, R2, P1, P2 to Undistort and Rectify Images
Correspondence – Dense or Sparse
Disparity Map
3D Reconstruction - Reprojection (Q)
Undistort and Rectify Points of Interest
3D Reconstruction – Triangulation
1 & 2
Rectified Points
Lens Distortion Correction
Correspondence
![Page 15: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/15.jpg)
Canonical Configuration – OpenCV Function
stereoRectify( ) returns an op/onal 4x4 reprojec/on matrix Q that can be used to reproject disparity map to 3D space.
For an array of points: void perspectiveTransform(const Mat& src, Mat& dst, const Mat& mtx); For an image: void reprojectImageTo3D( const Mat& disparity, Mat& 3dImage, const
Mat& Q, bool handleMissingValues=false ); This rou/ne takes a single‐channel disparity image and transforms
each pixel’s (x, y) coordinates along with that pixel’s disparity (i.e., a vector [x y d]) to the corresponding 3D point by using the 4‐by‐4 reprojec/on matrix Q. The output is a 3‐channel floa/ng‐point (or a 16‐bit integer) image of the same size as the input.
2/27/12 15 ECEn 631
![Page 16: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/16.jpg)
General Configuration – Triangulation 1
If the intrinsic and extrinsic parameters are known,3D points can be calculated from their projections, pl and pr.
The intrinsic and extrinsic parameters are estimated through calibration procedures.
If Tl,Rl , and Tr,Rr are the extrinsic parameters of the two cameras in the world coordinate system, the extrinsic parameters of the stereo system, T and R, are
2/27/12 16 ECEn 631
!
R = RrRlT
and T =Tr " RTl
![Page 17: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/17.jpg)
Another way to prove this relation.
2/27/12 17 ECEn 631
!
R = RrRlT
and T =Tr " RTl
i.e., Pr = RrP +Tr and Pl = RlP +Tl
and
Pl = RT
(Pr "T ) = (RrRlT
)T
(Pr "T )
= (RrRlT
)T
(RrP +Tr "T ) = (RrRlT
)T
(RrP + RTl )
= (RrRlT
)TRrP + (RrRl
T)TRTl ) = RlRr
TRrP + R
TRTl
= RlP +Tl
!
R = RrRlT
and T =Tl " RTTr
i.e., Pr = RrP +Tr and Pl = RlP +Tl
and
Pr = R(Pl "T ) = (RrRlT
)(Pl "T )
= (RrRlT
)(RlP +Tl "T ) = (RrRlT
)(RlP + RTTr )
= (RrRlT
)RlP + (RrRlT
)RTTr = RrRl
TRlP + RR
TTr
= RrP +Tr
Use right camera as reference
Use left camera as reference
![Page 18: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/18.jpg)
Another way to prove this relation.
2/27/12 18 ECEn 631
!
R = RrRlT
and T =Tr " RTl!
R = RrRlT
and T =Tl " RTTr
Right camera as reference
Left camera as reference
!
T1 =Tr " RTl T2 =Tl " RTTr
"RTT1 = "R
TTr + R
TRTl =Tl " R
TTr =T2
or
"RT2 = "RTl + RRTTr =Tr " RTl =T1
Rotation matrix is the same but translation vector differs.
![Page 19: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/19.jpg)
!
l = apl r = bRTpr " R
TT
!
apl + cw = apl + c(pl " RTpr ) = #R
TT +bR
Tpr
Ol (a=0)
Or (b=0)
pl
pr
P’
l r
w
Expressed in the left camera frame
2/27/12 19 ECEn 631
R and T relate right camera to left camera. OpenCV uses left camera are the reference.
![Page 20: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/20.jpg)
!
apl + cw = apl + c(pl " RT pr ) = #RTT +bRT pr
apl #bRTpr + c(pl " R
Tpr ) = #R
TT
There exists an unique solution of (a, b, and c) iff the two rays l and r are not parallel.
3D point P’ is the mid point of apl and -RTT+bRTpr
2/27/12 20 ECEn 631
![Page 21: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/21.jpg)
Implementation – with rectification
Use undistortPoints() to correct lens distor/on and rec/fica/on (if R and P are available from rec/fica/on) for corresponding points. undistortPoints(OldPoints, NewPoints, camMatrix, camDistort, R1/2, P1/2);
Data points in pixels can be obtained from rec/fied images directly.
Rec/fica/on could affect image quality
2/27/12 ECEn 631 21
![Page 22: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/22.jpg)
Implementation – w/o rectification Use undistortPoints() to correct lens distor/on but not rec/fica/on (if R and P are not available from rec/fica/on) for corresponding points. undistortPoints(OldPoints, NewPoints, camMatrix, camDistort);
undistortPoints() transforms data points in pixels into image frame before undistor/on but does not transform them back (bad).
Convert all points back to image frame – NewPoints.x = NewPoints.x fx + cx – NewPoints.x = NewPoints.x fy + cy
2/27/12 ECEn 631 22
![Page 23: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/23.jpg)
Implementation
2/27/12 ECEn 631 23 !
apl + cw = apl + c(pl " RTpr ) = #RTT +bR
Tpr
apl #bRT pr + c(pl " R
T pr ) = #RTT
a
fx 0 ox0 fy oy0 0 1
$
%
& &
'
(
) ) left
#1xlyl1
$
%
& &
'
(
) ) #b
r11 r12 r13r21 r22 r23r31 r32 r33
$
%
& &
'
(
) )
Tfx 0 ox0 fy oy0 0 1
$
%
& &
'
(
) ) right
#1xryr1
$
%
& &
'
(
) ) +
c
fx 0 ox0 fy oy0 0 1
$
%
& &
'
(
) ) left
#1xlyl1
$
%
& &
'
(
) ) "r11 r12 r13r21 r22 r23r31 r32 r33
$
%
& &
'
(
) )
Tfx 0 ox0 fy oy0 0 1
$
%
& &
'
(
) ) rightt
#1xryr1
$
%
& &
'
(
) )
*
+
, , ,
-
.
/ / /
= #RTTxTyTz
$
%
& &
'
(
) )
a
x
y
z
$
%
& &
'
(
) ) #b
x'
y'
z'
$
%
& &
'
(
) ) + c
x' '
y' '
z' '
$
%
& &
'
(
) )
=
x #x' x' '
y #y' y' '
z #z' z' '
$
%
& &
'
(
) )
a
b
c
$
%
& &
'
(
) )
= #r11 r12 r13r21 r22 r23r31 r32 r33
$
%
& &
'
(
) )
TTxTyTz
$
%
& &
'
(
) )
![Page 24: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/24.jpg)
Implementation Use solve(A, B, X, flag) to solve for the linear system AX=B.
The solu/on X is a vector of {a, b, c}. The 3D points measured from the leT camera can then be calculated as
2/27/12 ECEn 631 24
!
X
Y
Z
"
#
$ $
%
&
' '
=
(ax+bx' )2
(ay+by' )2
(az+bz' )2
"
#
$ $ $ $
%
&
' ' ' '
!
A =
x "x' x' '
y "y' y' '
z "z' z' '
#
$
% %
&
'
( ( X =
a
b
c
#
$
% %
&
'
( ( B = "
r11 r12 r13
r21 r22 r23
r31 r32 r33
#
$
% %
&
'
( (
TTxTyTz
#
$
% %
&
'
( (
![Page 25: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/25.jpg)
!
Recall Pr = R(Pl "T ) or
XrYrZr
#
$
% %
&
'
( (
= R(
XlYlZl
#
$
% %
&
'
( ( "TxTyTz
#
$
% %
&
'
( ( )
R, T : rotation and translation between two cameras
pl = MlPl = MlP
pr = MrPr = MrR(Pl "T ) = MrR(P "T )
2/27/12 25 ECEn 631
General Configuration – Triangulation 2
![Page 26: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/26.jpg)
!
! pl = flPl
Zl " xl = fl
Xl
Zl and yl = fl
Yl
Zl
xr = frXr
Zr= fr
R1(Pl #T )
R3(Pl #T )" xrR3(Pl #T ) = frR1(Pl #T )
Pl =( frR1 # xrR3)T
( frR1 # xrR3) " Zl = fl
Pl
pl= fl
( frR1 # xrR3)T
( frR1 # xrR3)pl
$Xl = xlZl fl Yl = ylZl fl
Can also calculate Pr = R(Pl #T )
2/27/12 26 ECEn 631
Implementation?
General Configuration – Triangulation 2
![Page 27: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/27.jpg)
3D Similarity (Affine) Reconstruction If only the intrinsic parameters are known,3D points can be calculated from their projections, pl and pr., up to a unknown scale factor.
The intrinsic parameters (Ml and Mr) can be estimated through calibration procedures. The relative position (R & T) of the cameras is unknown.
Less will be measured in an unknown scene compared to the full 3D reconstruction because the baseline (distance between the cameras) is unknown.
How to determine this factor? We can if we know the distance between two points in the observed scene or object.
2/27/12 27 ECEn 631
![Page 28: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/28.jpg)
1. Find correspondences between two images (pixels) and use the intrinsic parameters to convert them to be in the camera frames.
2. Estimate the essential matrix E using 8-point algorithms (can only be known up to an unknown scaling factor)
3. Obtain R and T from E 4. Use Euclidean full reconstruction to find 3D information up
to an unknown scaling factor.
0=lTr Epp
2/27/12 28 ECEn 631
Steps 1 and 2 can also be done by finding F first (easier because OpenCV has a findFundamentalMatrix function) and then
!
E = (Mr )TF Ml
!
p =
xcyc1
"
#
$ $
%
&
' '
=
fx 0 ox0 fy oy0 0 1
"
#
$ $
%
&
' '
(1xiyi1
"
#
$ $
%
&
' '
![Page 29: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/29.jpg)
Estimate T & R
Compute SVD of the new E=UDVT (svd() function)
Replace D with Σ to obtain the normalized E Calculate R and T as
!
RZT
(±"
2) =
0 ±1 0
!1 0 0
0 0 1
#
$
% %
&
'
( (
) =
* 0 0
0 * 0
0 0 0
#
$
% %
&
'
( (
=
1 0 0
0 1 0
0 0 0
#
$
% %
&
'
( (
ˆ T =
0 +Tz Ty
Tz 0 +Tx
Ty Tx 0
#
$
% %
&
'
( (
!
E = ˆ T R " R3#3Essential Matrix
2/27/12 29 ECEn 631 !
R = URz
T(±"
2)V
T
ˆ T = URz (±"
2)#U
T
The unknown scale factor
can be calculated if object size is
known.
Normalized
![Page 30: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/30.jpg)
Finding One Set of Unique Solution There are four solu/ons but only one yields geometrically consistent (i.e. posi/ve) Zl and Zr for all the points. 1. Obtain normalized E 2. Calculate all possible T and R 3. Reconstruct Zl and Zr using triangula/on (1 or 2). 4. Both nega/ve for some points, change the sign of T and
go back to step 3. 5. One posi/ve, one nega/ve for some point, change the
sign of each entry of the normalized E and go back to step 2
6. Both posi/ve for all points.
2/27/12 ECEn 631 30
![Page 31: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/31.jpg)
Reconstruction up to a Scale Factor
Since R and T are es/mated from normalized E, 3D reconstruc/on is only up to a scale factor.
The es/mated R and T can s/ll be used to rec/fy images.
All Euclidean full reconstruc/on methods can be used to reconstruct 3D.
2/27/12 ECEn 631 31
![Page 32: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/32.jpg)
3D Projective Reconstruction The most general 3D reconstruction case
If the point correspondence in two uncalibrated (both intrinsic and extrinsic parameters are unknown) cameras can be established, a 3D projective reconstruction can be obtained (only up to an unknown projective transformation) – Fundamental matrix is available but R and T cannot be obtained from F.
The reconstruction accuracy is only affected by the computation of disparities not by calibration.
The reconstructed coordinates X’ differ from the correct Euclidean reconstruction by some unknown projective transformation H,
X’ = HX
H is a regular 4x4 matrix. H depends on how much calibration knowledge is at hand.
2/27/12 32 ECEn 631
![Page 33: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/33.jpg)
!
x1
x2
x3
"
#
$ $ $
%
&
' ' '
= MintM
ext
Xw
Yw
Zw
1
"
#
$ $ $ $
%
&
' ' ' '
= M
Xw
Yw
Zw
1
"
#
$ $ $ $
%
&
' ' ' '
We call M a projective matrix
!
x1
x2
x3
"
#
$ $ $
%
&
' ' '
=
m1
T
m2
T
m3
T
"
#
$ $ $
%
&
' ' '
Xw
Yw
Zw
1
"
#
$ $ $ $
%
&
' ' ' '
=
m1
T
m2
T
m3
T
"
#
$ $ $
%
&
' ' '
P
!
x1: x
2: x
3= m
1
TP :m
2
TP :m
3
TP
2/27/12 33 ECEn 631
![Page 34: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/34.jpg)
!
x1m2
TP = x
2m1
TP
x1m3
TP = x
3m1
TP
x2m3
TP = x
3m2
TP
!
x1m2
T " x2m1
T
x1m3
T " x3m1
T
x2m3
T " x3m2
T
#
$
% % %
&
'
( ( (
P = 0
!
x1m3
T " x3m1
T
x2m3
T " x3m2
T
#
$ %
&
' ( P = 0
The first row and the second row are linearly dependent.
2/27/12 34 ECEn 631
![Page 35: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/35.jpg)
!
x1m3
T " x3m1
T
x2m3
T " x3m2
T
x1
'm'
3
T "x3
'm'
1
T
x2
'm'
3
T "x3
'm'
2
T
#
$
% % % %
&
'
( ( ( (
Pw
= APw
= 0
Correspondence is known between (x, y) and (x’, y’)
2/27/12 35 ECEn 631
![Page 36: ECEn631 15 - 3-D Reconstructionece631web.groups.et.byu.net/Lectures/ECEn631 15 - 3-D... · 2012. 2. 27. · stereoRectify( ) returns an oponal 4x4 reprojecon matrix Q that can be](https://reader035.vdocuments.net/reader035/viewer/2022081402/6079bcb677f55d6aa17d5162/html5/thumbnails/36.jpg)
PMHPHMPMyx
p
PMHPMHMPyx
p
r
l
~'~''1''
~~
1
1
1
==≅
=
==≅
=
−
−
2/27/12 36 ECEn 631