human computer interaction 1. overview and history of...
TRANSCRIPT
![Page 1: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/1.jpg)
No.3
ステレオ・エピポーラ幾何
Stereo and Epipolar Geometry
担当教員:向川康博・田中賢一郎
2019Computer Vison
![Page 2: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/2.jpg)
Slide credits
• Special Thanks: Some slides are adopted from other instructors’ slides.
– Tomokazu Sato, former NAIST CV1 class
– James Tompkin, Brown CSCI 1430 Fall 2017
– Ioannis Gkioulekas, CMU 16-385 Spring 2018
– We also thank many other instructors for sharing their slides.
2
![Page 3: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/3.jpg)
Today’s topics
3
• 3D point from matched point pair
• Epipolar geometry• E and F matrices• 8-point algorithm
• Other techniques• Multi-view (Stereo, EPI, Space curving)• Active illumination (SL,ToF)
![Page 4: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/4.jpg)
3D scene recovery
• Can you compute 3D scene from an image?
4
![Page 5: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/5.jpg)
Geometry of stereo views
• A single image does not have direct depth information.
• With at least two images, depth can be measured through triangulation.
– The most animals have at least two eyes.
– A stereo vision system is equipped on an autonomous robot as eyes.
5
?
?
![Page 6: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/6.jpg)
2-view geometry
• Image pixel corresponds to a ray in 3D space.
• Additional view gives the depth information.
6
ImageImage
Camera center
Which point is the real?
![Page 7: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/7.jpg)
Structure(scene geometry)
Motion(camera geometry)
Measurements
Pose Estimation known estimate3D to 2D
correspondences
Triangulation estimate known2D to 2D
correspondences
Reconstruction estimate estimate2D to 2D
correspondences
Triangulation(三角測量)
![Page 8: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/8.jpg)
Two views
8
Left Right
Tsukuba Stereo
mini-report1: What is the difference?
![Page 9: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/9.jpg)
Toy example
9
Disparity (視差): near object moves larger
baseline 𝑏C C’
image plane
𝑓 focal length 𝑓
𝑍
disparity 𝑑
If two cameras are parallel and have the same focal length,
𝑍 =𝑏𝑓
𝑑mini-report
2
left
right
disparities
objects
![Page 10: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/10.jpg)
(General) Triangulation Problem
10
image 2image 1
Given a matched points
And camera projection matrices
EstimateBest fit 3D point
𝑥, 𝑥′
𝑀,𝑀′𝑋
𝑋
![Page 11: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/11.jpg)
Recall: camera projection
11
3D points to 2D image points
𝒙 = 𝑴𝑿
camera center
image plane
principal axis
![Page 12: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/12.jpg)
Recall: Cross Product
Vector (cross) product takes two vectors and returns a vector perpendicular to both
cross product of two vectors in the same direction is zero
remember this!!!
![Page 13: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/13.jpg)
Solve for X
• Start from camera projection𝒙 = 𝑴𝑿
• Expand the right hand side
𝑴𝑿 =
−𝒎𝟏⊤ −
−𝒎𝟐⊤ −
−𝒎𝟑⊤ −
𝑿 =
𝒎𝟏⊤𝑿
𝒎𝟐⊤𝑿
𝒎𝟑⊤𝑿
• Same direction? cross product is zero
𝑥𝑦1×
𝒎𝟏⊤𝑿
𝒎𝟐⊤𝑿
𝒎𝟑⊤𝑿
=
𝑦𝒎𝟑⊤𝑿−𝒎𝟐
⊤𝑿
𝒎𝟏⊤𝑿 − 𝑥𝒎𝟑
⊤𝑿
𝑥𝒎𝟐⊤𝑿 − 𝑦𝒎𝟏
⊤𝑿
=000
13Number of unknown is 3 and equation is 2.
(Linear combination)
known known unknown
redundant
![Page 14: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/14.jpg)
Solve for X
• Same for the other cameraAnd superpose
𝑦𝒎𝟑⊤𝑿 −𝒎𝟐
⊤𝑿
𝒎𝟏⊤𝑿 − 𝑥𝒎𝟑
⊤𝑿
𝑦′𝒎′𝟑⊤𝑿 −𝒎′𝟐
⊤𝑿
𝒎′𝟏⊤𝑿 − 𝑥′𝒎′𝟑
⊤𝑿
=
0000
𝑨𝑿 = 𝟎• How to solve? SVD!
14
came from1st camera
came from2nd camera
Number of unknown is 3 and equation is 4.
![Page 15: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/15.jpg)
How to obtain the matched pair?
image 2image 1
𝑋
left right
![Page 16: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/16.jpg)
17
Naïve method: Template matching
For each point of the left image, find the best similar point from the right hand image.
Template
How many evaluation is required?
Width * height * width * height * [cost of template comparison]Example) 1000 * 1000 * 1000 * 1000 = 1012 times template comparison.
![Page 17: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/17.jpg)
Today’s topics
18
• 3D point from matched point pair
• Epipolar geometry• E and F matrices• 8-point algorithm
• Other techniques• Multi-view (Stereo, EPI, Space curving)• Active illumination (SL,ToF)
![Page 18: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/18.jpg)
Another camera
• One camera looks at this scene.
• How does another camera look?
19
Another cameraEpipole
Line corresponds to pointEpipolar line
![Page 19: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/19.jpg)
Epipolar geometry
• Where is the corresponding point?
• Search on the epipolar line.
20
ImageImageCamera center
1D search
![Page 20: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/20.jpg)
Epipolar geometry
Epipole(projection of o’ on the image plane)
Baseline
Epipolar plane
Image plane
Epipolar line(intersection of Epipolar plane
and image plane)
![Page 21: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/21.jpg)
Epipolar constraint
Potential matches for x lie on the epipolar line
![Page 22: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/22.jpg)
Examples
26
epipole
Converging (向かい合った) camera
![Page 23: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/23.jpg)
Calibrated case
• The intrinsic parameters of each camera are known.
• Essential matrix (基本行列): E (3x3 matrix)– Epipolar line
– Relationship between two corresponding points
27
![Page 24: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/24.jpg)
Parameters in essential matrix E
• Parameterized by
– the three DOF of the rotation matrix
– the two DOF of the direction of the translation vector (defined up to scale)
• Include only five independent parameters.
• The E can be decomposed into R and t.
28
Rt
MemoCross product can be expressed by
inner product using skew-symmetric
matrix (歪対称行列) such that
yxyx ][
𝐄 = 𝐑 𝒕×
mini-report
3
![Page 25: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/25.jpg)
Derivation of E
• coplanarity
• rigid motion
29
𝐑, 𝒕
𝒕
Three vectors (𝒙, 𝒕, 𝒙′) are coplanar.
𝒙 − 𝒕 ⊺ 𝒕 × 𝒙 = 0
𝒙′ = 𝐑(𝒙 − 𝒕)
𝒙′ = 𝐑 𝒙 − 𝒕
𝒙′⊺ = 𝒙 − 𝒕 ⊺𝐑⊺
𝒙′⊺𝐑 = 𝒙 − 𝒕 ⊺
𝒙′⊺𝐑 𝒕 × 𝒙 = 0
𝒙′⊺𝐑 𝒕× 𝒙 = 0
𝒙′⊺𝐑 𝒕× 𝒙 = 0
𝒙′⊺𝐄𝒙 = 0
![Page 26: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/26.jpg)
properties of the E matrix
Epipolar lines
Longuet-Higgins equation
Epipoles
(points in normalized camera coordinates)
![Page 27: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/27.jpg)
Uncalibrated case
• The intrinsic parameters K of each camera are unknown.
• The physical coordinates 𝒙 are unknown. Only the image coordinates 𝒑 are known.
• Fundamental matrix(基礎行列): F (3 x 3 matrix)
42
The Fundamental matrix is a generalization of the Essential matrix,
where the assumption of calibrated cameras is removed.
𝒑′⊺𝑭𝒑 = 0
𝒑 = 𝑲𝒙
![Page 28: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/28.jpg)
Fundamental Matrix
43
Recall: Intrinsic parameters K
𝒑 = 𝐊𝒙 𝒙 = 𝐊−𝟏𝒑
Recall: Essential matrix E
𝒙′⊤𝐄𝒙 = 0
Substitute
𝒑′⊤𝐊′−⊤𝐄𝐊−𝟏𝒑 = 0
= 𝐅
• Potential matches for lie on the epipolar line
Properties
𝒑 𝐅𝒑
• F is defined up to scale, and rank(F) = 2.
![Page 29: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/29.jpg)
Weak calibration
• The simple problem of estimating the epipolargeometry from point correspondences between two images with unknown intrinsic parameters.
• Corresponding pixel pairs
• Unknown 3x3 fundamental matrix
44
𝑭 =
𝑓1 𝑓2 𝑓3𝑓4 𝑓5 𝑓6𝑓7 𝑓8 𝑓9
𝑝𝑚′ = 𝑥′𝑚, 𝑦𝑚
′ , 𝑝𝑚= 𝑥𝑚, 𝑦𝑚
![Page 30: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/30.jpg)
One equation from one correspondence
• One correspondence
• Epipolar constraint
• One equation
45
𝑝𝑚′ = 𝑥′𝑚, 𝑦𝑚
′ , 𝑝𝑚= 𝑥𝑚, 𝑦𝑚
𝑝′𝑚⊺ 𝑭𝑝𝑚 = 0
![Page 31: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/31.jpg)
Eight-point algorithm
• F is 3x3 matrix (9 parameters)
• Since F is defined up to scale, we can set f9=1.
• 8 unknowns
• We need at least 8 points
• How to solve 𝑨𝒙 = 0? SVD!
46
MR4
![Page 32: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/32.jpg)
Example
![Page 33: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/33.jpg)
![Page 34: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/34.jpg)
line: 0.0295𝑥 + 0.9996𝑦 − 265.1531 = 0
![Page 35: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/35.jpg)
Where is the epipole?
How would you compute it?
![Page 36: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/36.jpg)
The epipole is in the right null space of F
S V D !
![Page 37: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/37.jpg)
Today’s topics
66
• What is 3D points recovery problem?
• 3D point from matched point pair
• Epipolar geometry• E and F matrices• 8-point algorithm
• Other techniques• Multi-view (Stereo, EPI, Space curving)• Active illumination (SL, ToF)
![Page 38: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/38.jpg)
Image rectification
• The two image planes are reprojected onto a common plane parallel to the base line.
• The rectified epipolar lines are scanlines of the new images, and they are also parallel to the base line.
68
x x'
o o'
![Page 39: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/39.jpg)
Ambiguity (曖昧性)
• When a single image feature is observed, there is no ambiguity.
• In the more usual case, wrong correspondences yield incorrect reconstructions.
69
x x'
o o'
X
o o'
?
![Page 40: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/40.jpg)
Three cameras (trinocular stereo)
• Adding a third camera eliminates the ambiguity
• The third image can be used to check hypothetical matches between the first two pictures.
– Colinear alignment: robust to occlusion
– Perpendicular alignment: robust for all directional edges
70
Colinear alignment Perpendicular alignment
![Page 41: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/41.jpg)
Multiple cameras
• Matches are found using all pictures
• Picking the first image as a reference, sums of squared differences associated with all other cameras are added into a global evaluation function.
• Robust to repetitive pattern
71
...
Combining multiple baseline stereo pairs
only one pair
combination of all cameras
![Page 42: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/42.jpg)
Epipolar plane image (EPI)
72
t
u
v
(a) Images taken under linear motion of camera with constant speed or by a light-field camera
(b) Space time volume (c) Epipolar plane image
Slope level of edge depends on the depth of the object.
![Page 43: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/43.jpg)
Space carving (視体積交差法)
73
*Kiriakos N. Kutulakos , Steven M. Seitz, A Theory of Shape by Space Carving, ICCV99
1. Make voxel space and put initial voxels2. Select one of surface voxel3. Check the photo-consistency with considering occlusions4. Remove voxel if photo-consistency is lower than threshold5. Repeat 2 to 5 until any voxel is not removed
*image of voxel from wiki
![Page 44: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/44.jpg)
Structured light
• Substitute a projector to a camera.
– Easy to obtain the correspondence.
• What pattern is used?
– Gray code, XOR, phase shifting, micro PS, etc.
74
Real-time structured light profilometry: a review
![Page 45: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/45.jpg)
Time-of-Flight
• Measures the delay of returning light.
76
𝜙
𝑎
𝑑 =𝑐𝜙
4𝜋𝑓
c: speed of light
f: frequency of modulation
https://blogs.msdn.microsoft.com/uk_faculty_connection/2017/05/15/image-based-motion-analysis-with-kinect-v2-and-opencv/
![Page 46: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/46.jpg)
Final mini report
• To obtain accurate 3d points by stereo, should the baseline be wide or narrow?
– How fine depth can be described by 1 pixel disparity?
– Are all pixels visible from both cameras? (occlusion problem: 隠れ問題)
– How about the difficulty of correspondence search?
78baseline
baseline
oo o'
o'
![Page 47: Human Computer Interaction 1. overview and history of HCIomilab.naist.jp/class/CV/2019/2019-CV-No3-Stereo-ForReport.pdf · ステレオ・エピポーラ幾何 Stereo and Epipolar](https://reader035.vdocuments.net/reader035/viewer/2022070815/5f0f02d17e708231d4420b7e/html5/thumbnails/47.jpg)
79
narrow baseline
wide baseline