image processing 4. optical flow taleix/opticalflow.pdfimage processing: 4. optical flow aleix m....
Post on 12-Jun-2020
11 Views
Preview:
TRANSCRIPT
1
Image Processing:4. Optical Flow4. Optical Flow
Aleix M. Martinezaleix@ece.osuedu
Motion estimation•Optical flow is used to compute the motion of the
pixels of an image sequence. It provides a dense(point to point) pixel correspondance.•Correspondence problem: determine where the
pixels of an image at time t are in the image attime t+1.•Large number of applications.
Two important definitions•Motion field:“the 2-D projection of a 3-Dmotion onto the image plane.”•Optical flow:“the apparent motion of the brightness pattern in an image sequence.”
The method ofThe method ofHorn and SchunckHorn and Schunck
•This is the most fundamental optical flow algorithm.•As you will see, it has several important flaws thatmakes its use inappropriate in a large number ofapplications.•Most of the other algorithms proposed to date arebased on the formulation advanced by Horn andSchunck.
2
•If the brightness is assumed to be constant fromframe to frame, then the motion associated toeach pixel (x,y) of an image I can be modeled as:
This is known as the data conservationconstraint.•The 1st-order Taylor expansion
),,(),,( tttvytuxIzyxI
0 tyx IvIuI
R
tyxD dxdyIvIuIE 2)(
),,(),,( tttvytuxIzyxI
tI
tyI
yxI
xtyxItyxI ),,(),,(
0t 0
tI
yI
dtdy
xI
dtdx
0dtdI
,dtdx
u ,dtdy
v ,xI
I x
,yI
I y
.tI
I t
0 tyx IvIuI
•Derivations of previous result:
•The method is underdetermined.•We can add an additional constraint known
as: spatial coherence
•The solution can be obtained by minimizingthe functional:
R
yxyxS dxdyvvuuE ))()(( 2222
R
S dxdyvuE 2),(
SD EE
Minimization
dxdyvvuuvuF yxyx ),,,,,(
To minimize the above integral, we can use calculus of variations.The Euler equations are:
0
yx uuu F
yF
xF
0
yx vvv F
yF
xF
22222tyxyxyx IvIuIvvuuF
And we want to minimize the expression:
xtyx IIvIuIu 2
ytyx IIvIuIv 2
Note that:
2
2
2
22
yx
(Details in: Horn, “Robot Vision”, MIT Press, 1986.)
The discrete case
We can estimate the derivatives, Ix andIy, by using the following discreetapproximation:
1,1,11,,1,1,1,,141
kjikjikjikjix IIIII
1,1,1,,,1,,,41
kjikjikjikji IIII
3
Example
Example
1
Limitations
•The assumptions embedded in Horn &Schunck formulation are generallyinappropriate.•E.g. both constraints are violated at motion
boundaries–known as motiondiscontinuities.•The aperture problem: to gain accuracy R
needs to be large, however, the larger R is themost probable that our assumptions becomeinvalid.
Robust Statistics:Robust Statistics:Black & AnandanBlack & Anandan
•It is possible to regard motion discontinuities asoutliers.•We can discard outliers only if we can detectthem.•Outliers normally deviate largely from the meanmotion.
Same constraints as in Horn & Schunck
R
tyxD dxdyIvIuIE )(
R
S dxdyvuE )),((
2
21
1log),(
x
x
New estimator, e.g.:
4
Examples
1
Illumination Changes:Illumination Changes:NegahdaripourNegahdaripour
•The methods seen above assume nochanges in the illumination of a scene fromframe to frame.•This is not realistic; even if theillumination source(s) is (are) not moving.•It is possible to modify the constraint usedby the two preceding method.
The illumination problem Illumination Changes
•Illumination changes also violate the assumptionsof ED and ES.
•B&A approach cannot handle large variations inlighting. Its formulation does not take this intoaccount.•We can easily incorporate this information in the
form of a multiplier and an offset:
),,(),,(),,(),,( tyxCtyxItyxMtttvytuxI
5
Gennert & Negahdaripour
R
tttyxD dxdycmIIvIuIE )(
R
S dxdyvuE 2
2),(
R
tM dxdymE2
2
R
tC dxdycE2
2
CCMMSSDD EEEEE
Closed solutionNegahdaripour has proposed the following closed-form solution:
W
t
t
ty
tx
W
y
x
y
x
y
y
y
yx
x
x
yx
I
II
II
II
cmyx
I
I
I
II
II
II
I
II
I
II
I
II
II
Ix
1
2
2
2
Images:
Horn & Schunck:
Ground-truth:
Images:
Black & Anandan:
Ground-truth:
Images:
Gennert & Negahdaripour:
Ground-truth:
More Examples
Images:
6
Horn & Schunck Gennert & Negahdaripour
Images Horn & Schunck
Black & Anandan Gennert & Negahdaripour
7
Error estimationError estimation
v
n
igtestimate
vel n
VVErr
v
ii
1
o
n
i gtestimate
gtestimate
ang n
VV
VV
Err
o
ii
ii
1
arccos
The flower sequence
Gaussian
Flower sequence
Vel. Err. Stdv. Density Angl. Err. Stdv. DensityH&S 1.388 0.662 100% 78.144 65.407 39.96%L&K 1.268 0.782 100.00% 73.061 64.264 62.43%N&Y 0.562 0.313 100% 18.652 21.086 99.32%B&A 0.531 0.386 100% 73.264 57.76 97.82%G&N 0.821 0.283 100% 14.696 16.226 98.97%
The Yosemite sequence
Yosemite sequence
Vel. Err. Stdv. Density Angl. Err. Stdv. DensityH&S 0.935 1.35 100% 45.342 85.14 98.58%L&K 0.746 1.159 99.88% 30.54 73.761 98.40%N&Y 0.636 0.726 99.97% 33.877 76.764 99.01%B&A 0.509 0.725 100% 18.775 57.809 99.13%G&N 1.547 1.786 100% 41.999 77.642 98.59%
Some Examples
8
9
top related