optical flow methods cisc 489/689 spring 2009 university of delaware
TRANSCRIPT
Optical Flow Methods
CISC 489/689Spring 2009
University of Delaware
Outline
• Review of Optical Flow Constraint, Lucas-Kanade, Horn and Schunck Methods
• Lucas-Kanade Meets Horn and Schunck• 3D Regularization• Techniques for solving optical flow• Confidence Measures in Optical Flow
Optical Flow Constraint
0
0
),,(),,(
),,(),,(
tyx fvfuf
t
f
dt
dy
y
f
dt
dx
x
f
tyxft
ft
y
fy
x
fxtyxf
tyxftttvytuxf
0limit takingand by Dividing tt
),( tvytux
),( yx
),,( tyxf
),,( ttyxf
Interpretation
Constraint Line
tT
tyx
fv
uf
fvfuf
0
yx ff ,
v
u 0
1
v
u
fff tyx
Lucas-Kanade Method
* = JK J
1
*),( v
u
JKvuELK
2
2
2
2
2
2
0
1
ttytx
tyyyx
txyxx
ttytx
tyyyx
txyxx
fffff
fffff
fffff
J
v
u
fffff
fffff
fffff
Lucas-Kanade Method
• Local Method, window based• Cannot solve for optical flow everywhere• Robust to noise
5.7 15
Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005
Dense optical Flow
?)(),( Minimize 2 tyx fvfufvuE
Lacks Smoothness
Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005
Horn and Schunck MethoddxdyvufvfufvuE tyxHS )()(),(
222
Euler-Lagrange Equations
Horn and Schunck Method
510 610Figures from Lucas/Kanade Meets Horn/Schunck: Combining Local and Global OpticFlow Methods ANDR´ES BRUHN AND JOACHIM WEICKERT, 2005
• Global Method • Estimates flow everywhere• Sensitive to noise• Oversmooths the edges
Why combine them?
• Need dense flow estimate• Robust to noise• Preserve discontinuities
Combining the two…
1
*),( v
u
JKvuELK
dxdyvufvfufvuE tyxHS )()(),(222
dxdyvuv
u
JKvuECLG )(
1
*),(22
Combined Local Global Method
dxdyvuv
u
JKvuECLG )(
1
*),(22
Euler-Lagrange Equations
AverageError
Standard Deviation
Lucas&Kanade 4.3(density 35%)
Horn&Schunk 9.8 16.2
Combining local and global 4.2 7.7
Table: Courtesy - Darya Frolova, Recent progress in optical flow
Preserving discontinuities• Gaussian Window does not preserve
discontinuities• Solutions
– Use bilateral filtering
– Add gradient constancy
dxdyvuv
u
JKvuE bilbil )(
1
*),(22
dxdyffvuv
u
JKvuE tgrad2
1
22)()(
1
*),(
Bilateral support window
Images: Courtesy, Darya Frolova, Recent progress in optical flow
Robust statistics – simple exampleFind “best” representative for the set of numbers
min2
iixxEL2: min
iixxEL1:
xix
Influence of xi on E: xi → xi + ∆
)mean( ixx
Outliers influence the most
ixx proportional to
xxEE ioldnew 2
)median( ixx
Majority decides
equal for all xi
oldnew EE
Slide: Courtesy - Darya Frolova, Recent progress in optical flow
Robust statistics
many ordinary people a very rich man
Oligarchy
Votes proportional to the wealth
Democracy
One vote per person
wealth
like in L2 norm minimization like in L1 norm minimization
Slide: Courtesy - Darya Frolova, Recent progress in optical flow
Combination of two flow constraints
robust: L1
x
robust in presence of outliers– non-smooth: hard to analyze
usual: L2
easy to analyze and minimize– sensitive to outliers
2x
robust regularized
smooth: easy to analyze robust in presence of outliers
22 x
ε
video
warpedwarped IIII min
),,( ; )1,,( tyxIItvyuxIIwarped
[A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performanceSlide: Courtesy - Darya Frolova, Recent progress in optical flow
Robust statistics
3D Regularization
• accounted for spatial regularization
• If velocities do not change suddenly with time, can we regularize in time as well?
)(22vu
3D Regularization
t
v
y
v
x
vv
t
u
y
u
x
uu
3
3
dxdydtvuv
u
JKvuETXCLG )(
1
*),(2
3
2
3],0[3
Multiresolution estimation
21
image Iimage J
Gaussian pyramid of image 1 Gaussian pyramid of image 2
Image 2image 1
run iterative estimation
run iterative estimation
warp & upsample
.
.
.
Multi-resolution Lucas Kanade Algorithm
Compute Iterative LK at highest level•For Each Level i• Take flow u(i-1), v(i-1) from level i-1
• Upsample the flow to create u*(i), v*(i) matrices of twice resolution for level i.
• Multiply u*(i), v*(i) by 2
• Compute It from a block displaced by u*(i), v*(i)
• Apply LK to get u’(i), v’(i) (the correction in flow)
• Add corrections u’(i), v’(i) to obtain the flow u(i), v(i) at the ith level, i.e., u(i)=u*(i)+u’(i), v(i)=v*(i)+v’(i)
Comparison of errors
For Yosemite sequence with cloudsTable: Courtesy - Darya Frolova, Recent progress in optical flow
Solving the system
fAu
How to solve?
2 components of success: fast convergence
good initial guess
Start with some initial guess
and apply some iterative method
initialu
Relaxation schemes have smoothing property:
Only neighboring pixels are coupled in relaxation
scheme
It may take thousands of iterations to propagate
information to large distance
. . . . . . . . . . . .
Relaxation smoothes the error
Relaxation smoothes the error Examples
2D case:
1D case:
Error of initial guess Error after 5 relaxation Error after 15 relaxations
Idea: coarser grid
On a coarser grid low frequencies become higher
Hence, relaxations can be more effective
initial grid – fine grid
coarse grid – we take every second point
Multigrid 2-Level V-Cycle
1. Iterate ⇒ error becomes smooth
2. Transfer error equation to the coarse level ⇒ low frequencies become high
3. Solve for the error on the coarse level ⇒ good error estimation
4. Transfer error to the fine level
5. Correct the previous solution6. Iterate ⇒ remove interpolation artifacts
make iteration process faster (on the coarse grid we can effectively minimize the error)
obtain better initial guess (solve directly on the coarsest grid)
Coarse grid - advantages
initialu
fAu
Coarsening allows:
go to the coarsest grid
solve here the equation
to findinitialu
interpolate to the
finer grid
initialu
Multigrid approach – Full scheme
Confidence Metric
• Intrinsic in Local Methods• How to evaluate for global methods?
– Edge strength?• Doesn’t work (Barron et al.,1994)
Confidence Metric
• Histogram of error contribution
Error
Number of pixels
Confidence Metric
More Results
More Results
Further Reading• Combining the advantages of local and global optic flow methods
(“Lucas/Kanade meets Horn/Schunck”) A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005
• High accuracy optical flow estimation based on a theory for warping
T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005• Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 -
2005• Towards ultimate motion estimation: Combining highest accuracy
with real-time performance. A. Bruhn, J. Weickert, 2005• Bilateral filtering-based optical flow estimation with occlusion
detection. J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006