1 recent progress in optical flow progress presented by: darya frolova and denis simakov
Post on 19-Dec-2015
236 Views
Preview:
TRANSCRIPT
1
Recent progress in optical flow
Recent progress in optical flow
progres
s
Presented by: Darya Frolova and Denis Simakov
2
Optical Flow is not in favorOptical Flow is not in favor
Often not using Optical Flow is stated as one of the main advantages of a method
Optical Flow methods have a reputation of either unreliable or slow
Very popular slide:
Optical Flow can be computed fast and accurately
Recent works claimRecent works claim:
3
Horn&Schunck
Optical Flow Research: TimelineOptical Flow Research: Timeline
Lucas&Kanade
1981 1992 1998 now
Benchmark:Barron et.al.
Benchmark:Galvin et.al.
many methods
more methods
no significant improvement, but a lot of useful ingredients were developed
Seminal papers
4
In This LectureIn This LectureWe will describe :
Ingredients for an accurate and robust optical flow
How people combine these ingredients
Fast algorithms
• Combining the advantages of local and global optic flow methods (“Lucas/Kanade meets Horn/Schunck”)
A. Bruhn, J. Weickert, C. Schnörr, 2002 - 2005• High accuracy optical flow estimation based on a theory for warping T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004 - 2005• Real-Time Optic Flow Computation with Variational Methods A. Bruhn, J. Weickert, C. Feddern, T. Kohlberger, C. Schnörr, 2003 -
2005• Towards ultimate motion estimation: Combining highest accuracy with real-time
performance A. Bruhn, J. Weickert, 2005
• Bilateral filtering-based optical flow estimation with occlusion detection J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006
Papers:
5
What is Optical Flow?What is Optical Flow?
…
6
DefinitionsDefinitions
The optical flowoptical flow is a velocity field in the image which transforms one image into the next image in a sequence [ [ Horn&SchunckHorn&Schunck ] ]
The motion field … is the projection into the image of three-dimensional motion vectors [ [ Horn&SchunckHorn&Schunck ]]
+ =
frame #1 frame #2flow field
7
Ambiguity of optical flowAmbiguity of optical flow
Frame 1Frame 1
flow (1): true motionflow (2)
8
Applications
• video compression• 3D reconstruction• segmentation• object detection• activity detection• key frame extraction• interpolation in time
optical flow
We are usually interested in actual motion
motion field
9
OutlineOutline
Ingredients for an accurate and robust optical flow
Local image constraints on motion
Robust statistics
Spatial coherence
How people combine these ingredients
Fast algorithms
10
Local image constraintsLocal image constraints
11
Brightness ConstancyBrightness Constancy
0),,()1,,( tyxItvyuxI
frame t
frame t+1
)1,,( tvyux
),,( tyx
v
u
12
),,()( tyx
)( )( )( )( )( tyxLIN IvIuI
1 v
uIII tyx
Complex dependence on
vu , Linearize:
Linearized brightness constancyLinearized brightness constancy
),,()1,,( tyxItvyuxI Deviation from brightness constancy (we want to minimize it)
13
Linearized brightness constancyLinearized brightness constancy
2
2
2
tytxt
tyyxy
txyxx
IIIII
IIIII
IIIII
J J – “motion tensor”, or “structure tensor”
1 v
uIII tyxLIN
Let us square the difference:
1 ]1 [
1 1 2 v
uJvuv
uIII
III
vu tyx
t
y
x
LIN
14
Averaged linearized constraintAveraged linearized constraint
1 ]1 [2 v
uJvuLIN
2
2
2
tytxt
tyyxy
txyxx
IIIII
IIIII
IIIII
J
J is a function of x, y (a matrix for every point)
J *= JG
Combine over small neighborhoods (more robust to noise):
15
Method of Lucas&KanadeMethod of Lucas&Kanade
1 ) ( ]1 [2 v
uJGvuGAUSSLIN
Solve independently for each point [ Lucas&Kanade 1981 ]
2min GAUSSLIN linear system
Can be solved for every point where matrix is not degenerate
01
2
2
v
u
IIIII
IIIIIG
tyyxy
txyxx
16
Lukas&Kanade - ResultsLukas&Kanade - Results
flow field flow field
Hamburg taxiRubik cube
17
Brightness is not always constantBrightness is not always constant
Rotating cylinder
Brightness constancy does not always hold
),,()1,,( tyxItvyuxI
inte
nsity
position
),,()1,,( tyxItvyuxI
Gradient constancy holds
inte
nsity
de
rivat
ive
position
18
Local constraints - SummaryLocal constraints - Summary
We have seen
• brightness constancy
0),,()1,,( tyxItvyuxI
linearized
averaged linearized
0),,()1,,( tyxItvyuxI
• gradient constancy
01
) ( ]1 [2
v
uJGvuGAUSSLIN
01
]1 [2
v
uJvuLIN
averaged linearized
01
) ( ]1 [2
v
uJGvuGAUSSLIN
19
Local constraints are not enough!
20
Local constraints work poorlyLocal constraints work poorly
input videoOptical flow direction using
only local constraints
color encodes direction as marked on the boundary
21
Where local constraints failWhere local constraints fail
Uniform regions
Motion is not observable in the image (locally)
22
Where local constraints failWhere local constraints fail
“Aperture problem”We can estimate only one flow component (normal)
23
Where local constraints failWhere local constraints fail
Occlusions
We have not seen where some points moved
Occluded regions are marked in red
24
Obtaining support from neighborsObtaining support from neighbors
Two main problems with local constraints:
• information about motion is missing in some points => need spatial coherency
• constraints do not hold everywhere => need methods to combine them robustly
good
missing
wrong
25
Robust combination of partially reliable data
or: How to hold elections
26
Toy exampleToy example
Find “best” representative for the set of numbers
min2
iixxEL2: min
iixxEL1:
xix
Influence of xi on E: xi → xi + ∆
)mean( ixx Outliers influence the most
ixx proportional to
xxEE ioldnew 2
)median( ixx Majority decides
equal for all xi
oldnew EE
27
Elections and robust statistics Elections and robust statistics
many ordinary people a very rich man
Oligarchy
Votes proportional to the wealth
Democracy
One vote per person
wealth
like in L2 norm minimization like in L1 norm minimization
28
Combination of two flow constraintsCombination of two flow constraints
robust: L1
x
robust in presence of outliers– non-smooth: hard to analyze
usual: L2
easy to analyze and minimize– sensitive to outliers
2x
robust regularized
smooth: easy to analyze robust in presence of outliers
22 x
ε
video
warpedwarped IIII min
),,( ; )1,,( tyxIItvyuxIIwarped
[A. Bruhn, J. Weickert, 2005] Towards ultimate motion estimation: Combining highest accuracy with real-time performance
29
Spatial PropagationSpatial Propagation
30
Obtaining support from neighborsObtaining support from neighbors
good
missing
wrong
Two main problems with local constraints:
• information about motion is missing in some points => need spatial coherency
• constraints do not hold everywhere => need methods to combine them robustly
31
Homogeneous propagationHomogeneous propagation
video
vu22
min- flow in the x direction- flow in the y direction- gradient
),,( tyxu),,( tyxv
[Horn&Schunck 1981]
This constraint is not correct on motion boundaries => over-smoothing of the resulting flow
32
Robustness to flow discontinuitiesRobustness to flow discontinuities
(also known as isotropic flow-driven regularization)
video
vu )||||(min 22
:
[T. Brox, A. Bruhn, N. Papenberg, J. Weickert, 2004]High accuracy optical flow estimation based on a theory for warping
22 x
ε
33
Selective flow filteringSelective flow filtering
[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006]Bilateral filtering-based optical flow estimation with occlusion detection
Solution: use “bilateral” filter in space, intensity, flow; taking into account occlusions
We want to propagate information
• without crossing image and flow discontinuities
• from “good” points only (not occluded)
34
Bilateral filterBilateral filter
x
I
x
I
x
I
x
I
Unilateral (usual) Bilateral
Preserves discontinuities!
[C. Tomasi, R. Manduchi, 1998] Bilateral filtering for gray and color images.
35
Using of bilateral filter - ExampleUsing of bilateral filter - Example
occluded regions
cyan rectangle moves to the right and occludes background region marked by red
[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, 2006]Bilateral filtering-based optical flow estimation with occlusion detection
36
Learning of spatial coherenceLearning of spatial coherence
• Come to the next lecture …
37
Spatial coherence: SummarySpatial coherence: Summary
Homogeneous propagation - oversmoothing
Robust statistics with homogeneous propagation - preserves flow discontinuities
Bilateral filtering - combines information from regions with similar flow and similar intensitiesHandles occlusions
38
Two more useful ingredientsTwo more useful ingredients
in brief – one slide each
2D vs. 3D2D vs. 3D
2 frames:
Several frames:
Several frames allow more accurate optical flow estimation
40
MultiscaleMultiscale Optical FlowOptical Flow
(other names: “warping”, “coarse-to-fine”, “multiresolution”)
pyramid for frame 1
tyx IvIuItyxItvyuxI ),,()1,,(
Linearization: valid only for small flow
+
+frame 1warped
?
pyramid for frame 2
upsample
41
MethodsMethods
How to make tasty soup with these ingredients: several recipes
42
OutlineOutline
Ingredients for an accurate and robust optical flow
How people combine these ingredients
Lukas & Kanade meet Horn & Schunck
The more ingredients – the better
Bilateral filtering and occlusions
Fast algorithms
43
Combining ingredientsCombining ingredients
Energy = ∫∫ϕϕ ((Data) + ∫∫ϕϕ ((“Smoothness”)
Local constraints–Brightness constancy
–Image gradient constancy
Spatial coherency–Homogeneous
–Flow-driven
–Bilateral filtering + occlusions
Combined using robust statistics Computed coarse-to-fine
Use several frames
44
Combining Local and GlobalCombining Local and Global
video
vu
JGvu1
) ( ]1 [
videovideo
vuvu
Jvu22
1
]1 [
videovideo
vuvu
Jvu22
1
)(G ]1 [
Lucas&Kanade Horn&Schunk
Basic “Combining local and global” [A. Bruhn, J. Weickert, C. Schnörr, 2002 ]
2
2
2
tytxt
tyyxy
txyxx
IIIII
IIIII
IIIII
JRemember:
45
Sensitivity to noise – quantitative resultsSensitivity to noise – quantitative results
ground truth flow
frame t+1
frame t
Error measure: angle between true and computed flow in
(x,y,t) space
46
The more ingredients - the betterThe more ingredients - the better
videovideo
warpedwarped IIII flow
brightness constancy
gradient constancy
spatial coherence
[Bruhn, Weickert, 2005]Towards ultimate motion estimation: Combining highest accuracy with real-time performance
47
Quantitative resultsQuantitative results
AverageStandard Deviation
Lucas&Kanade 4.3(density 35%)
Horn&Schunk 9.8 16.2
Combining local and global 4.2 7.7
“Towards ultimate …” 2.4 6.7
Angular error
Method
Yosemite sequence with clouds
Average error decreases, but standard deviation is still high….
48
Influence of each ingredientInfluence of each ingredient
For Yosemite sequence with clouds
49
Handling occlusionsHandling occlusions
videovideo
warped smoothnessII
bilateral filteringbilateral filtering of flow:preserve intensity and flow
discontinuities; model occlusions
:
[J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi, ECCV 2006] Bilateral filtering-based optical flow estimation with occlusion detection
50
Qualitative resultsQualitative results
51
Quantitative resultsQuantitative results
AverageStandard Deviation
Lucas&Kanade 4.3(density 35%)
Horn&Schunk 9.8 16.2
Combining local and global 4.2 7.7
“Towards ultimate …” 2.4 6.7
Bilateral+occlusionsBilateral+occlusions 2.62.6 6.16.1
Angular error
Method
Yosemite sequence with clouds
52
OutlineOutline
Ingredients for an accurate and robust optical flow
How people combine these ingredients
Fast algorithms
Energy functional => discrete equation
Multigrid solver: nearly real-time
53
How to minimize energyHow to minimize energy
dxuuxFuE )',,()(minimize
Necessary condition:
0'
uF
dxd
uF
Necessary condition
Analogy:
Euler-Lagrange equation
)(minimize xf
0)(' xf
54
An exampleAn example
Horn&Schunk
videovideo
vuvu
Jvu22
1
]1 [minimize
Let us see how to derive discretized equation for 1D Horn & Schuhck
image
ttxx uIuIIuIuE2222 ')2()(1D version
(simplified):
55
Iterative minimization Iterative minimization (simple example)(simple example)
0''2 uIIuI txxEuler-Lagrange
Discretized:
Local iterations: )( oldnewi ufu
Linear system of equation for u
02
211
h
uuubua iii
iii
2
2
2
2
211
ha
hb-
ha
uuu
i
i
i
oldi
oldinew
i
image
ttxx uIuIIuIuE2222 ')2()(
56
Life is not a picnicLife is not a picnic
Linear discretized system
image
utxItuxIuE )'( ),()1,( )( Even more complicated
Non-linear in u, non-linear
discretized system
image
ttxx uIuIIuIuE2222 ')2()(
0''2 uIIuI txx
image
ttxx uIuIIuIuE ')2()( 222
0))'('(2 uIIuI dxd
txx
57
Optimization algorithmsOptimization algorithms
Simple iterative minimization
Multigrid: much faster convergence
58
Solving the systemSolving the system
fAu
How to solve?
2 components of success: fast convergence
good initial guess
Start with some initial guess
and apply some iterative method
initialu
59
Relaxation schemes have smoothingsmoothing property:
Only neighboring pixels are coupled in relaxation scheme
It may take thousands of iterations to
propagate information to large distance
. . . . . . . . . . . .
Relaxation smoothes the errorRelaxation smoothes the error
60
Relaxation smoothes the error Relaxation smoothes the error ExamplesExamples
2D case:
1D case:
Error of initial guess Error after 5 relaxation Error after 15 relaxations
61
Idea: coarser gridIdea: coarser grid
On a coarser grid low frequencies become higher
Hence, relaxations can be more effective
initial grid – fine grid
coarse grid – we take every second point
62
Multigrid 2-Level V-CycleMultigrid 2-Level V-Cycle
1. Iterate ⇒ error becomes smooth
2. Transfer error equation to the coarse level ⇒ low frequencies become high
3. Solve for the error on the coarse level ⇒ good error estimation
4. Transfer error to the fine level
5. Correct the previous solution6. Iterate ⇒ remove interpolation artifacts
63
make iteration process faster (on the coarse grid we can effectively minimize the error)
obtain better initial guess (solve directly on the coarsest grid)
Coarse grid - advantagesCoarse grid - advantages
initialu
fAu
Coarsening allows:
go to the coarsest grid
solve here the equation
to findinitialu
interpolate to the
finer grid
initialu
64
Multigrid approach – Full schemeMultigrid approach – Full scheme
65
Non-linear: Full Approximation SchemeNon-linear: Full Approximation Scheme
⇒ Need to transfer current solution to the coarser level
A(·) non-linear
Equation for error involves current solution u0:
Difference from the linear case:
0
0
0 solutioncurrent
Aufr
uu
u
fuA )(
fuA )(fAu
rA fuA )( 0
Linear Non-linear
fine level:
coarse level:
66
Multigrid: SummaryMultigrid: Summary
• Used to solve linear or non-linear equations
• Method: combine two techniques– Basic iterative solver: quickly removes high
frequencies of the error– Coarsening: makes low frequencies high
• Contribution: fast minimization of loosely coupled equations
fAu fuA )(
67
Fast Optimization: ResultsFast Optimization: Results
Gauss-Seidel 1.15 0.87
Full multigrid 0.016 62.8
Gauss-Seidel 9.52 0.11
FAS-multigrid 0.087 11.5
Gauss-Seidel 34.5 0.03
FAS-multigrid 0.396 2.5
Horn&Schunck
CLG
“Towards ultimate…”
Time [sec] frames/sec
68
Summary of the TalkSummary of the Talk• 25 years of Optical Flow : a lot of useful ingredients were developed:
• local constraints:• brightness constancy• gradient constancy
• smoothing techniques:• homogeneous• flow-driven (preserving discontinuities)• bilateral filters• handling of occlusions
• robust functions • multiscale
• All ingredients are combined an a global Energy Minimization approach
• This difficult global optimization can be done very fast using Multigrid
69
Thank you!
70
Coarse grid - advantages
)4(1 2hO
Another advantages of using coarse grids:
• Relaxations are cheaper (because we have less poins) (1/2 as many points in 1D, 1/4 in 2D, 1/8 in 3D)
• Convergence rate is better ( instead of ))(1 2hO
71
Definition of the errorDefinition of the error
fAu true
After each iteration sweep we have a solution: fAu found
There are two measures of the correctness of the solution :
The error: foundtrue uue
The residual:foundAufr
rAe
) )( ( AvfAefAeAvevAAu
foundu
72
Some Approaches to Optical FlowSome Approaches to Optical Flow
),,(),,(),,(),,()1,,( tyxIvtyxIutyxItyxItvyuxI tyx
Gradient-based techniques
use spatiotemporal partial derivatives to estimate flow at each point
frame 1 frame 2
Patch comparison techniquespatch comparison
+ ++ u v
– shown to perform poorly
good performance– aperture problem
Image-drivenImage-driven
isotropic image-driven
2
2
22
1I
+
v+u
image gradient
Idea: relax smoothing constraint on strong image gradients
74
Anisotropic image-driven
anisotropic image-driven
encourages smooth flow along edges and not across
22
2
2
1
I
IID
vDvuDuT
TT
2 IuT
Intuition: penalize only along-the-edge non-smoothness of flow:
Actually used more complex:
75
Flow-driven regularizationFlow-driven regularization
anisotropic flow-driven
)()( vuvideo
video
ss )()( 21
[Aubert, Deriche, Kornprobst, 1995 ]
[Weickert, Schnorr 2000]
an alternative (one of):
• data terms• smoothness terms• complete functionals
76
Flow-driven regularizationFlow-driven regularization
anisotropic flow-drivenisotropic flow-driven
less penalty for non-smooth flow less penalty across flow edges
• data terms• smoothness terms• complete functionals
77
We need smart smoothingWe need smart smoothing
homogeneous
image-driven isotropic
image-driven anisotropic
flow-driven isotropic
flow-driven anisotropic
method
deg
ree
of s
moo
thn
ess
optimal level ?
• data terms• smoothness terms• complete functionals
78
What we getWhat we get
homogeneous
isotropic image-driven
anisotropic image-driven
isotropic flow-driven
anisotropic flow-driven
• data terms• smoothness terms• complete functionals
79
What we payWhat we pay
hom
og
en
eou
s
an
isotr
op
ic im
ag
e-d
riven
isotr
op
ic fl
ow
-dri
ven
isotr
op
ic im
ag
e-d
riven
an
isotr
op
ic fl
ow
-dri
ven
regularization (smoothness) methods
run
nin
g t
ime
60 sec
40 sec
20 sec
(imag
e si
ze 1
60x1
20)
• data terms• smoothness terms• complete functionals
80
min )( uE
u )(uE
energy functional
Discretized equation
Solve by
linear non-linear
local iterations Multigrid (simple for linear, FAS for non-linear)
equation
is a min for
satisfies Euler-Lagrange equation (ELE)
uhence
Energy minimization: overviewEnergy minimization: overview
81
Gauss-Seidel relaxationsGauss-Seidel relaxations
2
2
2
2
211
ha
hb-
ha
uuu
i
i
i
oldi
oldinew
i
• start from random initial guess
• solve for one point assuming its neighbors are fixed (Gauss-Seidel)
initialu
02
211
h
uuubua iii
iii
top related