media re-expression for stereo cinema
DESCRIPTION
Media Re-expression for Stereo Cinema. Félix Raimbault , François Pitié and Anil Kokaram. Overview 1/2. Motion Magnification for Stereo Videos Stereo Video Inpainting. Overview 2/2. Automatic Cartoonization Stereo Video Segmentation. Segmentation of a video volume - PowerPoint PPT PresentationTRANSCRIPT
1
Media Re-expressionfor Stereo Cinema
Félix Raimbault, François Pitié and Anil Kokaram
2
Overview 1/2• Motion Magnification for Stereo Videos
• Stereo Video Inpainting
3
Overview 2/2• Automatic Cartoonization
• Stereo Video SegmentationSegmentation of a
video volume[Collomosse et al.
2005]
4
Stereo Video Inpainting
5
Context• Initial motivation: fill-in
revealed areas for stereo motion magnification
• Correct artefacts arising during shooting in stereo
• Remove unwanted objects
• Fill-in dis-occluded areas fordisparity remapping
play video
6
Exemplar-based Framework: Priority• Priority: [Criminisi, A. et al. 2004]
• process first pixels with more available information nearby
• try to reconstruct first areas with “structure” (edges and depth discontinuity)
edge map
initial priority
order of filling
reconstruction
target frame
7
Exemplar-based Framework: Matching• Patch-matching strategy: [Efros, A. and Leung,
T. 1999]• find similar neighbourhood to current missing pixel
• SSIM-based distance: compare patches structure
...
I(x) replaced by I(x)
S(x) best match at site x ^
target patch around x
^
8
Patch Tracking• [Raimbault, F.
and Kokaram, A. SPIE 2011]
• use long-term data
• motion vector reconstruction inside the hole
• use data across views
• disparity vector reconstruction inside the hole
9
Smoothness• Spatial smoothness: (to be submitted to
WIAMIS’12)• “coherent patch sewing”• estimate average distance of selected patches as
a criteria to prune patch copying
missing data
pixel by pixel
coherence sewing
target frame
10
Smoothness• Stereo-Temporal smoothness: (to be submitted to
WIAMIS’12)• preferentially select patch in previously
reconstructed frame• stereo-spatio-temporal patch-matching
• de-activate smoothness for outliers (bad motion and disparity estimates)
t t+1t-1
v
v’
11
Luminance Correction• [Raimbault, F. and Kokaram, A. SPIE
2011]:• colour discrepancy• due to sampling from frames far
away in time from current frame (lighting can change)
• colour correction needed
• 1-tap linear predictive model:
• weighted least squares solution
12
Results• video “spywalk”• reconstruction of twist in the lash of the girl’s bag
• average SSIM: 0.9997 slightly better than Rig Removal: 0.9989
• small hole -> block matching is enough to estimate offsets
• video “water_drop”• “fresh” data input from other view in our technique
whereas reconstruction with Rig Removal degrades
• video “walking_girl”• block matching and pixel accuracy -> not enough
• Published in SPIE’11– Accepted for publication in JEI’12
play video
13
Issues• Issues:• greedy => lack of global coherence• more accurate motion vectors needed• parameters choice can be complicated (patch size,
search size)• lack of temporal smoothness
• To be explored:• experiment with global optimisation (QPBO)• use trajectories (Viterbi tracker)• patch stitching• impose constraints by first doing stereo-video
segmentation
14
Plan• Internship for Sony Research in Stuttgart• temporal stabilisation of videos
• Stereo Video Segmentation• based on [Baugh, G. and Kokaram, A. 2010]
• Return to Stereo Video Inpainting• feature point trajectories [Baugh, G. and Kokaram, A.
2009]