motion denoising with application to time-lapse...
Post on 21-Jul-2020
8 Views
Preview:
TRANSCRIPT
Michael Rubinstein MIT CSAIL
Motion Denoising with Application to Time-lapse Photography
Ce Liu Microsoft Research NE
Peter Sand Fredo Durand MIT
Bill Freeman MIT
Time-lapse Videos
Construction
Natural phenomena Medical Biological/Botanical
For Personal Use Too!
Source: YouTube
9 months
7 years
16 years
http://www.danhanna.com/aging_project/p.html
“Stylized Jerkiness”
Motion Denoising
Time
World Time-lapse
Space
Motion denoising
Motion Denoising
Motion
denoising
• Video summarization (video time-lapse)
• Time-lapse editing
Time-lapse in Vision/Graphics Research
[Bennett and McMillan 2007] [Pritch et al. 2008]
[Sunkavalli et al. 2007]
• Naïve low-pass (temporal) filtering
– Pixels of different objects are averaged
• Smoothing motion trajectories
– Motion estimation in time-lapse videos is hard!
* Motion discontinuities
* Color inconsistencies
Motion Denoising is Challenging!
KLT tracks
• Key idea: long-term events in videos can be statistically explained within some local spatiotemporal support, while short-term events are more distinctive
– Assumption: world is smooth
– Short-term variation = noise, long-term variation = signal
• Our algorithm reshuffles the pixels in both space and time to maintain long-term events in the video, while removing short-term noisy motions
Formulation
𝐸 𝑤 = |𝐼 𝑝 + 𝑤(𝑝) − 𝐼(𝑝
𝑝
)|
+ 𝛼 𝐼(𝑝 + 𝑤 𝑝 ) − 𝐼 𝑟 +𝑤(𝑟) 2
𝑝,𝑟∈𝑁𝑡(𝑝)
+ 𝛾 𝜆𝑝𝑞|𝑤 𝑝 −𝑤 𝑞 |
𝑝,𝑞∈𝑁(𝑝)
Formulation
𝑝 = (𝑥, 𝑦, 𝑡) 𝐼 – input video, 𝐼(𝑝 + 𝑤 𝑝 ) – output video
𝑁𝑡 𝑝 - Temporal neighbors of 𝑝, 𝑁 𝑝 - Spatiotemporal neighbors of 𝑝 𝑤 𝑝 ∈ 𝛿𝑥 , 𝛿𝑦 , 𝛿𝑡 : |𝛿𝑥| ≤ Δ𝑠, 𝛿𝑦 ≤ Δ𝑠 , 𝛿𝑡 ≤ Δ𝑡 - displacement field
𝜆𝑝𝑞 = exp −𝛽 𝐼 𝑝 − 𝐼 𝑞2 , 𝛽 = 2 𝐼 𝑝 − 𝐼 𝑞 2 −1
Fidelity (to input)
Temporal coherence
(of the result)
Regularization
(of the warp)
• Optimized discretely on a 3D MRF
– Nodes represent pixels
– state space of each pixel = volume of possible spatiotemporal shifts
• Complicated (huge!) inference problem
– E.g. 5003 nodes, 103 states per node
– Optimize using Loopy Belief Propagation
Optimization
• Potential functions message passing
– Message structure stored on disk; read and write message chunks on need
𝜓𝑝 𝑤 𝑝 = 𝐼 𝑝 + 𝑤 𝑝 − 𝐼 𝑝
𝜓𝑝𝑟𝑡 𝑤 𝑝 ,𝑤 𝑟 = 𝛼 𝐼 𝑝 +𝑤 𝑝 − 𝐼 𝑟 +𝑤 𝑟
𝟐+
𝛾𝜆𝑝𝑟|𝑤 𝑝 − 𝑤 𝑟 |
𝜓𝑝𝑞𝑡 𝑤 𝑝 ,𝑤 𝑞 = 𝛾𝜆𝑝𝑞|𝑤 𝑝 − 𝑤 𝑞 |
Optimization
Linear in state space +
Pre-compute
Quadratic in state space
(non convex)
Quadratic in state space
But can be computed in linear
time (distance transforms)
• Spatiotemporal video pyramid
– Smooth spatially
– Sample temporally
• Displacements in the coarse level used as centers for the search volume in the finer level
Multi-scale Processing
Results
x
y
future
past
Comparing with Other Optimization Techniques
Results
x
y
future
past
Results
Comparison with Naïve Temporal Filtering
x
t
Support Size
Motion-scale Decomposition
Motion-scale Decomposition
Other Scenarios
• User-controlled motion scales
– Not necessarily binary decomposition into long-term and short-term
• Modify the time-lapse capturing process to help post-processing
– E.g. use short videos instead of still images and find best “path” through the video
• Explore motion-denoising with time-lapse from other domains
– Embryos research, satellite imagery
Future Work
http://csail.mit.edu/mrub/timelapse
Thank you!
top related