3d imaging motion. extract visual information from image sequences assume illumination conditions...
TRANSCRIPT
3D IMAGING
Motion
Motion
Extract visual information from image sequences
Assume illumination conditions do not change
Differences in image Camera moves (Some) Objects in a scene move Both camera and objects move
Difference from stereo
Stereo from multiple views Images taken at the same time moment
Motion Images taken in small time intervals
Differences Camera position might not be known Changes are smaller Not necessary caused by a single 3D rigid transform
Rotation and Translation Multiple objects
Why analyze motion
Understand scene structure Objects, Distance, Velocities
Understand scene properties without a priori knowledge on the world
Random Dot
Time to Collision
f
L
v
L
Do
l(t)
An object of height L moves with constant velocity v:•At time t=0 the object is at:• D(0) = Do
•At time t it is at • D(t) = Do – vt
•It will crash with the camera at time:• D(t) = Do – vt = 0
• t = Do/v
t=0t
D(t)
Time to Collision
f
L
v
L
Do
l(t)
t=0t
D(t)
The image of the object has size l(t):
Taking derivative wrt time:
Time to Collision
f
L
v
L
Do
l(t)
t=0t
D(t)
And their ratio is:
Time to Collision
f
L
v
L
Do
l(t)
t=0t
D(t)
And time to collision:
Can be directly measured from image
Can be found, without knowing L or Do or v !!
Parallax
Parallax=apparent change in position when viewed from different lines of sights
Closer objects have larger parallax
Depth cues
Kinetic Depth Effect
Motion Field
3D velocity field Each point has a velocity vector
2D Motion field Projection of these 3D velocity vector onto
image plane
2D Motion Field
2D Optical Flow
Apparent motion of image brightness pattern
Assuming that illumination does not change:
Image changes are due to the RELATIVE MOTION between the scene and the camera.
There are 3 possibilities: Camera still, moving scene Moving camera, still scene Moving camera, moving scene
Motion Analysis Problems
Correspondence Problem Track corresponding elements across frames
Reconstruction Problem Given a number of corresponding elements, and
camera parameters, what can we say about the 3D motion and structure of the observed scene?
Segmentation Problem What are the regions of the image plane which
correspond to different moving objects?
Motion Field (MF)
The MF assigns a velocity vector to each pixel in the image.
These velocities are INDUCED by the RELATIVE MOTION btw the camera and the 3D scene
The MF can be thought as the projection of the 3D velocities on the image plane.
2D Motion Field and 2D Optical Flow Motion field: projection of 3D motion vectors on image plane
• Optical flow field: apparent motion of brightness patterns
• We equate motion field with optical flow field
0 0
00
Object point has velocity , induces in imagei
ii
P
d d
dt dt
v v
r rv v
2 Cases Where this Assumption Clearly is not Valid
(a) (b)
(a) A smooth sphere is rotating under constant illumination. Thus the optical flow field is zero, but the motion field is not.
(b) A fixed sphere is illuminated by a moving source—the shading of the image changes. Thus the motion field is zero, but the optical flow field is not.
What is Meant by Apparent Motion of Brightness Pattern?
The apparent motion of brightness patterns is an awkward concept. It is not easy to decide which point P' on a contour C' of constant brightness in the second image corresponds to a particular point P on the corresponding contour C in the first image.
Aperture Problem
(a) Line feature observed through a small aperture at time t.
(b) At time t+t the feature has moved to a new position. It is not possible to determine exactly where each point has moved. From local image measurements only the flow component perpendicular to the line feature can be computed.
Normal flow: Component of flow perpendicular to line feature.
(a) (b)
Brightness Constancy Equation Let P be a moving point in 3D:
At time t, P has coords (X(t),Y(t),Z(t)) Let p=(x(t),y(t)) be the coords. of its image
at time t. Let E(x(t),y(t),t) be the brightness at p at
time t. Brightness Constancy Assumption:
As P moves over time, E(x(t),y(t),t) remains constant.
Brightness Constraint Equation
flow. optical of components the, ,, and irradiance thebe ,,Let yxvyxutyxE
expansionTaylor
,,,, tyxEtttvytuxE
0limit takingand by dividing
,,,,
tt
tyxEet
Et
y
Ey
x
ExtyxE
0
derivative total theofexpansion theiswhich
0
dt
dE
t
E
dt
dy
y
E
dt
dx
x
E
short: 0 tyx EvEuE
Brightness Constancy Equation
Taking derivative wrt time:
Brightness Constancy Equation
Let(Frame spatial gradient)
(optical flow)
and(derivative across frames)
Brightness Constancy Equation
Becomes:
The OF is CONSTRAINED to be on a line !
Normal Motion/Aperture Problem
Barber Pole Illusion
Solving the aperture problem How to get more equations for a pixel?
Basic idea: impose additional constraints most common is to assume that the flow field is
smooth locally one method: pretend the pixel’s neighbors have the
same (u,v) If we use a 5x5 window, that gives us 25 equations per
pixel!
Constant flow Prob: we have more equations than unknowns
– The summations are over all pixels in the K x K window
• Solution: solve least squares problem
– minimum least squares solution given by solution (in d) of:
Taking a closer look at (ATA)
This is the same matrix we used for corner detection!
Taking a closer look at (ATA)
The matrix for corner detection:
is singular (not invertible) when det(ATA) = 0
One e.v. = 0 -> no corner, just an edgeTwo e.v. = 0 -> no corner, homogeneous region
Aperture Problem !
But det(ATA) = Õ li = 0 -> one or both e.v. are 0
A Pattern of Hajime Ouchi
Copyright A.Kitaoka 2003
Reconstruction
Rotation
P
rrr
rrr
rrr
3,32,31,3
3,22,21,2
3,12,11,1 Represents a 3D rotation of the points in P.
First, look at 2D rotation (easier)
n
n
yyy
xxx
21
21 ...
cossin
sincos
cossin
sincosR
Matrix R acts on points by rotating them.
• Also, RRT = Identity. RT is also a rotation matrix, in the opposite direction to R.
Simple 3D Rotation
n
n
n
zzz
yyy
xxx
21
21
21...
100
0cossin
0sincos
Rotation about z axis.
Rotates x,y coordinates. Leaves z coordinates fixed.
Full 3D Rotation
cossin0
sincos0
001
cos0sin
010
sin0cos
100
0cossin
0sincos
R
• Any rotation can be expressed as combination of three rotations about three axes.
100
010
001TRR
• Rows (and columns) of R are orthonormal vectors.
• R has determinant 1 (not -1).
Alper Yilmaz, Fall 2004 UCF
3D Motion
Displacement model Velocity model
Z
Y
X
TZYXZ
TZYXY
TZYXX
Z
Y
X
T
T
T
Z
Y
X
Z
Y
X
Z
Y
X
1
1
1
TR
Displacement Model and Its Projection onto Image Space
Z
Y
X
TZYXZ
TZYXY
TZYXX
Perspective projection
ZT
yx
ZT
yxy
ZT
yx
ZT
yxx
Z
Y
Z
X
1
1
x X Y Z TX
X Y Z TZ
y X Y Z TY
X Y Z TZ
Alper Yilmaz, Fall 2004 UCF
Velocity Model in 3DOptical Flow
Z
Y
X
T
T
T
Z
Y
X
Z
Y
X
1
1
1
Z
Y
X
T
T
T
Z
Y
X
ZZ
YY
XX
0
0
0
Z
Y
X
T
T
T
Z
Y
X
Z
Y
X
100
010
001
0
0
0
Alper Yilmaz, Fall 2004 UCF
Velocity Model in 3DOptical Flow
3
2
1
12
13
23
0
0
0
V
V
V
Z
Y
X
Z
Y
X
321
213
132
VXYZ
VZXY
VYZX
Z
Y
X
T
T
T
Z
Y
X
ZZ
YY
XX
0
0
0
rotational velocities
translational velocities
3D optical flow
VXX
3
2
1
Z
Y
X
X
cross product
Velocity Model in 2D
Perspective projection
321
213
132
VXYZ
VZXY
VYZX
Z
Yfy
Z
Xfx
yv
xu
212331
2
2213
32
1
)(
)(
yf
xyf
yZ
Vx
Z
Vfv
xf
xyf
yxZ
V
Z
Vfu