introduction to 3d vision how do we obtain 3d image data? what can we do with it?
TRANSCRIPT
![Page 1: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/1.jpg)
Introduction to 3D Vision
• How do we obtain 3D image data?
• What can we do with it?
![Page 2: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/2.jpg)
What can you determine about 1. the sizes of objects 2. the distances of objects from the camera?
What knowledgedo you use toanalyze this image?
![Page 3: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/3.jpg)
What objects are shown in this image?How can you estimate distance from the camera?What feature changes with distance?
![Page 4: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/4.jpg)
3D Shape from X
• shading• silhouette• texture
• stereo • light striping• motion
mainly research
used in practice
![Page 5: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/5.jpg)
Perspective Imaging Model: 1D
xi
xf
f
This is the axis of the real image plane.
O O is the center of projection.
This is the axis of the frontimage plane, which we use.zc
xcxi xc f zc
=
camera lens
3D objectpoint
B
D
E
image of pointB in front image
real imagepoint
![Page 6: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/6.jpg)
Perspective in 2D(Simplified)
P=(xc,yc,zc) =(xw,yw,zw)
3D object point
xc
yc
zw=zc
yi
Yc
Xc
Zc
xiF f
cameraP´=(xi,yi,f)
xi xc f zc
yi yc f zc
=
=
xi = (f/zc)xcyi = (f/zc)yc
Here camera coordinatesequal world coordinates.
opticalaxis
ray
![Page 7: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/7.jpg)
3D from Stereo
left image right image
3D point
disparity: the difference in image location of the same 3Dpoint when projected under perspective to two different cameras.
d = xleft - xright
![Page 8: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/8.jpg)
Depth Perception from StereoSimple Model: Parallel Optic Axes
f
f
L
R
camera
baselinecamera
b
P=(x,z)
Z
X
image plane
xl
xr
z
z xf xl
=
x-b
z x-bf xr
= z y yf yl yr
= =y-axis is
perpendicularto the page.
![Page 9: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/9.jpg)
Resultant Depth Calculation
For stereo cameras with parallel optical axes, focal length f,baseline b, corresponding image points (xl,yl) and (xr,yr)with disparity d:
z = f*b / (xl - xr) = f*b/d
x = xl*z/f or b + xr*z/f
y = yl*z/f or yr*z/f
This method ofdetermining depthfrom disparity is called triangulation.
![Page 10: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/10.jpg)
Finding Correspondences
• If the correspondence is correct, triangulation works VERY well.
• But correspondence finding is not perfectly solved. for the general stereo problem.
• For some very specific applications, it can be solved for those specific kind of images, e.g. windshield of a car.
° °
![Page 11: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/11.jpg)
3 Main Matching Methods
1. Cross correlation using small windows.
2. Symbolic feature matching, usually using segments/corners.
3. Use the newer interest operators, ie. SIFT.
dense
sparse
sparse
![Page 12: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/12.jpg)
Epipolar Geometry Constraint:1. Normal Pair of Images
x
y1
y2
z1 z2
C1 C2b
P
P1P2
epipolarplane
The epipolar plane cuts through the image plane(s)forming 2 epipolar lines.
The match for P1 (or P2) in the other image, must lie on the same epipolar line.
![Page 13: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/13.jpg)
Epipolar Geometry:General Case
P
P1P2
y1y2
x1
x2e1
e2
C1
C2
![Page 14: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/14.jpg)
Constraints
P
e1e2
C1
C2
1. Epipolar Constraint: Matching points lie on corresponding epipolar lines.
2. Ordering Constraint: Usually in the same order across the lines.
Q
![Page 15: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/15.jpg)
Structured Light
3D data can also be derived using
• a single camera
• a light source that can produce stripe(s) on the 3D object
lightsource
camera
light stripe
![Page 16: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/16.jpg)
Structured Light3D Computation
3D data can also be derived using
• a single camera
• a light source that can produce stripe(s) on the 3D object
lightsource
x axisf
(x´,y´,f)
3D point(x, y, z)
b
b[x y z] = --------------- [x´ y´ f] f cot - x´
(0,0,0)
3D image
![Page 17: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/17.jpg)
Depth from Multiple Light Stripes
What are these objects?
![Page 18: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/18.jpg)
Our (former) System4-camera light-striping stereo
projector
rotationtable
cameras
3Dobject
![Page 19: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/19.jpg)
Camera Model: Recall there are 5 Different Frames of Reference
• Object
• World
• Camera
• Real Image
• Pixel Image
yc
xc
zc
zwC
Wyw
xw
A
a
xf
yf
xpyp
zppyramidobject
image
![Page 20: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/20.jpg)
Rigid Body Transformations in 3D
zw
Wyw
xw
instance of the object in the
world
pyramid modelin its own
model space
xp
yp
zp
rotatetranslatescale
![Page 21: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/21.jpg)
Translation and Scaling in 3D
![Page 22: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/22.jpg)
Rotation in 3D is about an axis
x
y
z
rotation by angle
about the x axis
Px´Py´Pz´1
1 0 0 00 cos - sin 00 sin cos 00 0 0 1
PxPyPz1
=
![Page 23: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/23.jpg)
Rotation about Arbitrary Axis
Px´Py´Pz´1
r11 r12 r13 txr21 r22 r23 tyr31 r32 r33 tz0 0 0 1
PxPyPz1
=
T R1 R2
One translation and two rotations to line it up with amajor axis. Now rotate it about that axis. Then applythe reverse transformations (R2, R1, T) to move it back.
![Page 24: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/24.jpg)
The Camera Model
How do we get an image point IP from a world point P?
c11 c12 c13 c14c21 c22 c23 c24
c31 c32 c33 1
s Iprs Ipc
s
PxPyPz1
=
imagepoint
camera matrix C worldpoint
What’s in C?
![Page 25: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/25.jpg)
The camera model handles the rigid body transformation from world coordinates to camera coordinates plus the
perspective transformation to image coordinates.
1. CP = T R WP2. IP = (f) CP
s Ipxs Ipy
s
1 0 0 00 1 0 00 0 1/f 1
CPxCPyCPz
1
=
perspectivetransformation
imagepoint
3D point incamera
coordinates
![Page 26: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/26.jpg)
Camera Calibration
• In order work in 3D, we need to know the parameters of the particular camera setup.
• Solving for the camera parameters is called calibration.
yw
xwzw
W
yc
xc
zc
C
• intrinsic parameters are of the camera device
• extrinsic parameters are where the camera sits in the world
![Page 27: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/27.jpg)
Intrinsic Parameters
• principal point (u0,v0)
• scale factors (dx,dy)
• aspect ratio distortion factor
• focal length f
• lens distortion factor (models radial lens distortion)
C
(u0,v0)
f
![Page 28: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/28.jpg)
Extrinsic Parameters
• translation parameters t = [tx ty tz]
• rotation matrix
r11 r12 r13 0r21 r22 r23 0r31 r32 r33 00 0 0 1
R = Are there reallynine parameters?
![Page 29: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/29.jpg)
Calibration Object
The idea is to snapimages at differentdepths and get alot of 2D-3D pointcorrespondences.
![Page 30: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/30.jpg)
The Tsai Procedure
• The Tsai procedure was developed by Roger Tsai at IBM Research and is most widely used.
• Several images are taken of the calibration object yielding point correspondences at different distances.
• Tsai’s algorithm requires n > 5 correspondences
{(xi, yi, zi), (ui, vi)) | i = 1,…,n}
between (real) image points and 3D points.
![Page 31: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/31.jpg)
Tsai’s Geometric Setup
Oc
x
y
principal point p0
z
pi = (ui,vi)
Pi = (xi,yi,zi)
(0,0,zi)
x
y
image plane
camera
3D point
![Page 32: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/32.jpg)
Tsai’s Procedure
• Given n point correspondences ((xi,yi,zi), (ui,vi))
• Estimates • 9 rotation matrix values• 3 translation matrix values• focal length• lens distortion factor
• By solving several systems of equations
![Page 33: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/33.jpg)
We use them for general stereo.
P
P1=(r1,c1)P2=(r2,c2)
y1y2
x1
x2e1
e2
C1
C2
![Page 34: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/34.jpg)
For a correspondence (r1,c1) inimage 1 to (r2,c2) in image 2:
1. Both cameras were calibrated. Both camera matrices are then known. From the two camera equations we get 4 linear equations in 3 unknowns.
r1 = (b11 - b31*r1)x + (b12 - b32*r1)y + (b13-b33*r1)zc1 = (b21 - b31*c1)x + (b22 - b32*c1)y + (b23-b33*c1)z
r2 = (c11 - c31*r2)x + (c12 - c32*r2)y + (c13 - c33*r2)zc2 = (c21 - c31*c2)x + (c22 - c32*c2)y + (c23 - c33*c2)z
Direct solution uses 3 equations, won’t give reliable results.
![Page 35: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/35.jpg)
Solve by computing the closestapproach of the two skew rays.
If the raysintersectedperfectly in 3D,the intersectionwould be P.
V
Instead, we solve for the shortest linesegment connecting the two rays andlet P be its midpoint.
P1
Q1
Psolve forshortest
![Page 36: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/36.jpg)
Application: Kari Pulli’s Reconstruction of 3D Objects from light-striping stereo.
![Page 37: Introduction to 3D Vision How do we obtain 3D image data? What can we do with it?](https://reader035.vdocuments.net/reader035/viewer/2022062308/56649dbe5503460f94ab1193/html5/thumbnails/37.jpg)
Application: Zhenrong Qian’s 3D Blood Vessel Reconstruction from Visible Human Data