msu cse 240 fall 2003 stockman cv: 3d to 2d mathematics perspective transformation; camera...
Post on 20-Dec-2015
218 views
TRANSCRIPT
MSU CSE 240 Fall 2003 Stockman
CV: 3D to 2D mathematics
Perspective transformation; camera calibration;stereo computation;
and more
MSU CSE 240 Fall 2003 Stockman
Roadmap of topics Review perspective transformation Camera calibration Stereo methods Structured light methods Depth-from-focus Shape-from-shading
MSU CSE 240 Fall 2003 Stockman
Review coordinate systems
World or global W
Object or model M
Camera or sensor C
Camera or sensor D
MSU CSE 240 Fall 2003 Stockman
Convenient notation for points and transformations
This point P has 3 real world coordinates in coordinate system W
This point P has 2 real coordinates in the image
This transformation maps each point in the real world W to a point in the image I
MSU CSE 240 Fall 2003 Stockman
Current goal
Develop the theory in terms of modules (components) so that
concepts are understood and can be put into practical application
MSU CSE 240 Fall 2003 Stockman
Perspective transformation
X and Y are scaled by the ratio of focal length to depth Z
Camera origin is center of projection, not lens
MSU CSE 240 Fall 2003 Stockman
In next homework & project
fit camera model to image with jig jig has known precise 3D
coordinates examine accuracy of camera model use camera model to do graphics use two camera models to compute
depth from stereo
MSU CSE 240 Fall 2003 Stockman
Notes on perspective trans. 3D world scaled according to ratio of
depth to focal length scaling formulas are in terms of real
numbers with the same units e.g. mm in the 3D world and mm in the image plane real image coordinates must be further
scaled to pixel row and column entire 3D ray images to the same 2D
point
MSU CSE 240 Fall 2003 Stockman
Goal: General perspective trans to be developed (accept for now)
Camera matrix C transforms 3D real world point into image row and column using 11 parameters
MSU CSE 240 Fall 2003 Stockman
The 11 parameters Cij model internal camera parameters: focal length f ratio of pixel height and width any shear due to sensor chip alignment external orientation parameters: rotation of camera frame relative to world frame translation of camera frame relative to world
The 11 parameters of this model are NOT independent.
Radial distortion is not linear and is not modeled.
MSU CSE 240 Fall 2003 Stockman
Camera matrix via least squares
Minimize the residuals in the image plane. Get 2 equations for each pair ((r, c), (x, y, z))
MSU CSE 240 Fall 2003 Stockman
2 equations for each pair
Here, (u, v) is the point in the image where 3D point (x,y,z) is projected. The 11 unknowns d jk form the camera matrix.
Known image points
Known 3D points
Camera parameters
MSU CSE 240 Fall 2003 Stockman
2n linear equations from n pairs ((u,v) (x,y,z))
Standard linear algebra problem; easily solved in Matlab or by using a linear algebra package.
Often, package replaces b’s with the residuals.
MSU CSE 240 Fall 2003 Stockman
Use a jig for calibration Jig has known set
of points Measure points in
world system W or use the jig to define W
Take image with camera and determine 2D points
Get pairings ((r, c) (x, y, z))
MSU CSE 240 Fall 2003 Stockman
Example calibration data# # IMAGE: g1view1.ras# # INPUT DATA | OUTPUT DATA# |Point Image 2-D (U,V) 3-D Coordinates (X,Y,Z) | 2-D Fit Data Residuals X Y | A 95.00 336.00 0.00 0.00 0.00 | 94.53 337.89 | 0.47 -1.89 B 0.00 6.00 0.00 | | C 11.00 6.00 0.00 | | D 592.00 368.00 11.00 0.00 0.00 | 592.21 368.36 | -0.21 -0.36 N 501.00 363.00 9.00 0.00 0.00 | 501.16 362.78 | -0.16 0.22 O 467.00 279.00 8.25 0.00 -1.81 | 468.35 281.09 | -1.35 -2.09 P 224.00 266.00 2.75 0.00 -1.81 | 224.06 266.43 | -0.06 -0.43 # CALIBRATION MATRIX 44.84 29.80 -5.504 94.53 2.518 42.24 40.79 337.9 -0.0006832 0.06489 -0.01027 1.000
MSU CSE 240 Fall 2003 Stockman
3D points on jig
Dimensions in inches
MSU CSE 240 Fall 2003 Stockman
Jig set in workspace
Mapping is established between 3D points (x, y, z) and image points (u, v)
MSU CSE 240 Fall 2003 Stockman
Other jigs used at MSU frame with wires and beads placed in
car instead of the driver seat (to do stereo measurements of car driver)
frame with wires and beads as big as a harp to calibrate space for people walking (up to 6 cameras, persons wear tight body suit with reflecting disks, cameras compute 3D motion trajectory)
MSU CSE 240 Fall 2003 Stockman
Least squares set up
2n x 11
11 x 1 2n
x 1
=
A X = B
MSU CSE 240 Fall 2003 Stockman
Least squares abstraction
MSU CSE 240 Fall 2003 Stockman
Justify the form of camera matrix Another sequence of slides Rotation, scaling, shear in 3D real
world as a 3x3 (or 4x4) matrix Projection to real 2D image as 4x4
matrix Scaling real image coordinates to [r,
c] coordinates as 4x4 matrix Combine them all into one 4x4
matrix
MSU CSE 240 Fall 2003 Stockman
Other mathematical models
Two camera stereo
MSU CSE 240 Fall 2003 Stockman
Baseline stereo: carefully aligned cameras
MSU CSE 240 Fall 2003 Stockman
Computing (x, y, z) in 3D from corresponding 2D image points
MSU CSE 240 Fall 2003 Stockman
2 calibrated cameras view the same 3D point at (r1,c1)(r2,c2)
MSU CSE 240 Fall 2003 Stockman
Compute closest approach of the two rays: use center point V
Shortest line segment between rays
MSU CSE 240 Fall 2003 Stockman
Connector is perpendicular to both imaging rays
MSU CSE 240 Fall 2003 Stockman
Solve for the endpoints of the connector
Scaler mult. Fix book
MSU CSE 240 Fall 2003 Stockman
Correspondence problem: more difficult aspect
MSU CSE 240 Fall 2003 Stockman
Correspondence problem is difficult
Can use interest points and cross correlation
Can limit search to epipolar line Can use symbolic matching (Ch
11) to determine corresponding points (called structural stereopsis)
apparently humans don’t need it
MSU CSE 240 Fall 2003 Stockman
Epipolar constraint
With aligned cameras, search for corresponding point is 1D along corresponding row of other camera.
MSU CSE 240 Fall 2003 Stockman
Epipolar constraint for non baseline stereo computation
If cameras are not aligned, a 1D search can still be determined for the corresponding point. P1, C1, C2 determine a plane that cuts image I2 in a line: P2 will be on that line.
Need to know relative orientation of cameras C1 and C2
MSU CSE 240 Fall 2003 Stockman
Measuring driver body position
4 cameras were used to measure driver position and posture while driving: 2mm accuracy achieved