lucas-kanade optical flo16385/s15/lectures/lecture21.pdf · to obtain the least squares solution...
TRANSCRIPT
Lucas-Kanade Optical FlowComputer Vision 16-385
Carnegie Mellon University (Kris Kitani)
spatial derivative optical flow temporal derivative
Ix
u+ Iy
v + It
= 0
I
x
=@I
@x
Iy =@I
@y u =dx
dt
v =dy
dtIt =
@I
@t
How can we use the brightness constancy equation to estimate the optical flow?
Ix
u+ Iy
v + It
= 0
known
unknown
We need at least ____ equations to solve for 2 unknowns.
Ix
u+ Iy
v + It
= 0
known
unknown
Where do we get more equations (constraints)?
Ix
u+ Iy
v + It
= 0
Where do we get more equations (constraints)?
Assume that the surrounding patch (say 5x5) has constant flow
Flow is locally smooth
Neighboring pixels have same displacement
Using a 5 x 5 image patch, gives us 25 equations
Assumptions:
Flow is locally smooth
Neighboring pixels have same displacement
Using a 5 x 5 image patch, gives us 25 equations
Assumptions:
Ix
(p1)u+ Iy
(p1)v = �It
(p1)
Ix
(p2)u+ Iy
(p2)v = �It
(p2)
...
Ix
(p25)u+ Iy
(p25)v = �It
(p25)
Flow is locally smooth
Neighboring pixels have same displacement
Using a 5 x 5 image patch, gives us 25 equations2
6664
Ix
(p1) Iy
(p1)Ix
(p2) Iy
(p2)...
...Ix
(p25) Iy
(p25)
3
7775
uv
�= �
2
6664
It
(p1)It
(p2)...
It
(p25)
3
7775
Assumptions:
Matrix form
Flow is locally smooth
Neighboring pixels have same displacement
Using a 5 x 5 image patch, gives us 25 equations2
6664
Ix
(p1) Iy
(p1)Ix
(p2) Iy
(p2)...
...Ix
(p25) Iy
(p25)
3
7775
uv
�= �
2
6664
It
(p1)It
(p2)...
It
(p25)
3
7775
Ax
b25⇥ 2 2⇥ 1 25⇥ 1
How many equations? How many unknowns? How do we solve this?
Assumptions:
Least squares approximation
x = argminx
||Ax� b||2A
>Ax = A
>b
is equivalent to solving
To obtain the least squares solution solve:
Least squares approximation
x = argminx
||Ax� b||2A
>Ax = A
>b
is equivalent to solving
2
4
Pp2P
Ix
Ix
Pp2P
Ix
Iy
Pp2P
Iy
Ix
Pp2P
Iy
Iy
3
5
uv
�= �
2
4
Pp2P
Ix
It
Pp2P
Iy
It
3
5
where the summation is over each pixel p in patch P
A>A A>bx
x = (A>A)�1
A
>b
To obtain the least squares solution solve:
Least squares approximation
x = argminx
||Ax� b||2A
>Ax = A
>b
is equivalent to solving
2
4
Pp2P
Ix
Ix
Pp2P
Ix
Iy
Pp2P
Iy
Ix
Pp2P
Iy
Iy
3
5
uv
�= �
2
4
Pp2P
Ix
It
Pp2P
Iy
It
3
5
where the summation is over each pixel p in patch P
A>A A>bx
Sometimes called ‘Lucas-Kanade Optical Flow’(special case of the LK method with a translational warp model)
A
>Ax = A
>b
When is this solvable?
ATA should be invertible
ATA should not be too small
ATA should be well conditioned
λ1 and λ2 should not be too small
λ1/λ2 should not be too large (λ1=larger eigenvalue)
2
4
Pp2P
Ix
Ix
Pp2P
Ix
Iy
Pp2P
Iy
Ix
Pp2P
Iy
Iy
3
5
Where have you seen this before?
A>A =
2
4
Pp2P
Ix
Ix
Pp2P
Ix
Iy
Pp2P
Iy
Ix
Pp2P
Iy
Iy
3
5
Where have you seen this before?
Harris Corner Detector!
Implications• Corners are when λ1, λ2 are big; this is also when
Lucas-Kanade optical flow works best
• Corners are regions with two different directions of gradient (at least)
• Corners are good places to compute flow!
What happens when you have no ‘corners’?
You want to compute optical flow. What happens if the image patch contains only a line?
������������
In which direction is the line moving?
small visible image patch
������������
In which direction is the line moving?
small visible image patch
������������
������������
������������
������������
Want patches with different gradients to the avoid aperture problem
Want patches with different gradients to the avoid aperture problem
1 1 1 12 2 2 23 3 3 34 4 4 45 5 5 5
- - - -- 1 1 1- 2 2 2- 3 3 3- 4 4 4
y
x
y
x
optical flow: (1,1)
H(x,y) = y I(x,y)
It(3, 3) = I(3, 3)�H(3, 3) = �1
Ix
u+ Iy
v + It
= 0
Iy(3, 3) = 1
Ix
(3, 3) = 0v = 1
We recover the v of the optical flow but not the u. This is the aperture problem.
Compute gradients Solution:
Applied to optical flow: Assume that the flow over a small image patch is constant
Note: The Lucas-Kanade method is a general
framework for image matching (not only optical flow)
Horn-Shunck!Optical Flow
(1981)
Lucas-Kanade!Optical Flow!
(1981)
‘constant’ flow‘smooth’ flow
Constraint: Flow of neighboring pixels
should be smooth
Contraint: Flow over a small patch
should be the same
Horn-Schunck16-385 Computer Vision
Carnegie Mellon University (Kris Kitani)
Horn-Schunck !Optical Flow (1981)
Lucas-Kanade!Optical Flow (1981)
‘constant’ flow!(flow is constant for all pixels)
‘smooth’ flow!(flow can vary from pixel to pixel)
brightness constancymethod of differences
global method (dense)
local method (sparse)
small motion
Optical Flow (Problem definition)
Estimate the motion (flow) between these
two consecutive images
I(x, y, t)I(x, y, t0)
How is this different from estimating a 2D transform?
Key Assumptions (unique to optical flow)
Color Constancy!(Brightness constancy for intensity images)
Small Motion!(pixels only move a little bit)
Implication: allows for pixel to pixel comparison (not image features)
Implication: linearization of the brightness constancy constraint
Approach
I(x, y, t)I(x, y, t0)
Look for nearby pixels with the same color(small motion) (color constancy)
Brightness constancy
I(x, y, t)
(x, y)
(x+ u�t, y + v�t)
(x, y)
(u, v) (�x, �y) = (u�t, v�t)Optical flow (velocities): Displacement:
I(x+ u�t, y + v�t, t+ �t) = I(x, y, t)
I(x, y, t+ �t)
For a really small time step…
Optical Flow Constraint equationI(x+ u�t, y + v�t, t+ �t) = I(x, y, t)
I(x, y, t) +@I
@x
�x+@I
@y
�y +@I
@t
�t = I(x, y, t)
Ix
u+ Iy
v + It
= 0
@I
@x
dx
dt
+@I
@y
dy
dt
+@I
@t
= 0
Ix
u+ Iy
v + It
= 0
u
v
Solution lies on a straight line
The solution cannot be determined uniquely with a single constraint (a single pixel)
Where can we get an additional constraint?
Horn-Schunck !Optical Flow (1981)
Lucas-Kanade!Optical Flow (1981)
‘constant’ flow!(flow is constant for all pixels)
‘smooth’ flow!(flow can vary from pixel to pixel)
brightness constancymethod of differences
global method local method
small motion
Smoothness
most objects in the world are rigid or deform elastically !
moving together coherently
we expect optical flow fields to be smooth
Key idea (of Horn-Schunck optical flow)
Enforce brightness constancy
Enforce smooth flow field
to compute optical flow
Enforce brightness constancy
Ix
u+ Iy
v + It
= 0
minu,v
Ix
uij
+ Iy
vij
+ It
�2For every pixel,
Enforce brightness constancy
Ix
u+ Iy
v + It
= 0
minu,v
Ix
uij
+ Iy
vij
+ It
�2For every pixel,
lazy notation for Ix
(i, j)
Ix
Iy
changes these
displacement of object
u v changes these
It
+ =
Find the optical flow such that it satisfies:
Enforce smooth flow field
uij ui+1,jui�1,j
ui,j+1
ui,j�1
minu
(ui,j � ui+1,j)2
u-component of flow
Which flow field is more likely?
X
ij
(uij � ui+1,j)2
X
ij
(uij � ui+1,j)2?
Which flow field is more likely?
more likelyless likely
optical flow objective function
Optical flow constraint
Smoothness constraint
(uij � ui+1,j)
ij
i, j + 1
i+ 1, j ij
i, j + 1
i+ 1, ji� 1, j
i, j � 1
i� 1, j
i, j � 1
(uij � ui,j+1)
ij
i, j + 1
i+ 1, j ij
i, j + 1
i+ 1, ji� 1, j
i, j � 1
i� 1, j
i, j � 1
(vij � vi+1,j) (vij � vi,j+1)
Ed
(i, j) =
Ix
uij
+ Iy
vij
+ It
�2
Es(i, j) =1
4
(uij � ui+1,j)
2 + (uij � ui,j+1)2 + (vij � vi+1,j)
2 + (vij � vi,j+1)2
�
Horn-Schunck optical flow
minu, v
X
ij
⇢Ed(i, j) + �Es(i, j)
�
Ed
(i, j) =
Ix
uij
+ Iy
vij
+ It
�2
where
and
Es(i, j) =1
4
(uij � ui+1,j)
2 + (uij � ui,j+1)2 + (vij � vi+1,j)
2 + (vij � vi,j+1)2
�
How do we solve this minimization problem?
minu, v
X
ij
⇢Ed(i, j) + �Es(i, j)
�
How do we solve this minimization problem?
minu, v
X
ij
⇢Ed(i, j) + �Es(i, j)
�
Compute partial derivative, derive update equations (gradient decent)
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
it’s not so bad…
how many terms depend on k and l?
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
it’s not so bad…
how many terms depend on k and l?
Compute the partial derivatives of this huge sum!
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
(uij � ui+1,j)
ij
i, j + 1
i+ 1, ji� 1, j
i, j � 1
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
uij =1
4
⇢ui+1,j + ui�1,j + ui,j+1 + ui,j�1
�local average
(u2ij � 2uijui+1,j + u2
i+1,j) (u2ij � 2uijui,j+1 + u2
i,j+1)
(variable will appear four times in sum)
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
Ax = b
how do you solve this?
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
this is a linear system
ok, take a step back, why are we doing all this math?
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
We are solving for the optical flow (u,v) given two constraints
smoothness brightness constancy
We need the math to minimize this (back to the math)
@E
@vkl
= 2(vkl
� vkl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Iy
@E
@ukl
= 2(ukl
� ukl
) + 2�(Ix
ukl
+ Iy
vkl
+ It
)Ix
Where are the extrema of E?
(set derivatives to zero and solve for unknowns u and v)
Ax = b
how do you solve this?
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
Partial derivatives of Horn-Schunck objective function E:
x = A�1b =
adjA
detAb
Recall
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
x = A�1b =
adjA
detAb
Recall
(determinant of A)
�Ix
Iy
ukl
+ (1 + �I2y
)vkl
= vkl
� �Iy
It
(1 + �I2x
)ukl
+ �Ix
Iy
vkl
= ukl
� �Ix
It
(determinant of A)
Same as the linear system:{1 + �(I2
x
+ I2y
)}ukl
= (1 + �I2x
)ukl
� �Ix
Iy
vkl
� �Ix
It
{1 + �(I2x
+ I2y
)}vkl
= (1 + �I2y
)vkl
� �Ix
Iy
ukl
� �Iy
It
(determinant of A)
(determinant of A)
Rearrange to get update equations:
ukl
= ukl
� Ix
ukl
+ Iy
vkl
+ It
1 + �(I2x
+ I2y
)Ix
vkl
= vkl
� Ix
ukl
+ Iy
vkl
+ It
1 + �(I2x
+ I2y
)Iy
new value
old average
{1 + �(I2x
+ I2y
)}ukl
= (1 + �I2x
)ukl
� �Ix
Iy
vkl
� �Ix
It
{1 + �(I2x
+ I2y
)}vkl
= (1 + �I2y
)vkl
� �Ix
Iy
ukl
� �Iy
It
check for missing lambda
ok, take a step back, why did we do all this math?
X
ij
(1
4
(u
ij
� ui+1,j)
2 + (uij
� ui,j+1)
2 + (vij
� vi+1,j)
2 + (vij
� vi,j+1)
2
�+ �
Ix
uij
+ Iy
vij
+ It
�2)
We are solving for the optical flow (u,v) given two constraints
smoothness brightness constancy
We needed the math to minimize this (now to the algorithm)
Horn-Schunck Optical Flow Algorithm
1. Precompute image gradients!
2. Precompute temporal gradients!
3. Initialize flow field!
4. While not converged Compute flow field updates for each pixel:
ukl
= ukl
� Ix
ukl
+ Iy
vkl
+ It
1 + �(I2x
+ I2y
)Ix
vkl
= vkl
� Ix
ukl
+ Iy
vkl
+ It
1 + �(I2x
+ I2y
)Iy
Ix
Iy
It
u = 0
v = 0