matting and compositingcyy/courses/vfx/14spring/...the great train robbery (1903) matte shot optical...
TRANSCRIPT
Matting and Compositing
Digital Visual Effectsg ffYung-Yu Chuang
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
Photomontage Photomontage
The Two Ways of Life, 1857, Oscar Gustav Rejlander y jPrinted from the original 32 wet collodion negatives.
Photographic compositionsPhotographic compositions
Lang Ching-shan
Use of mattes for compositingUse of mattes for compositing
The Great Train Robbery (1903) matte shot
Use of mattes for compositingUse of mattes for compositing
The Great Train Robbery (1903) matte shot
Optical compositingOptical compositing
King Kong (1933) Stop-motion + optical compositing
Digital matting and compositingDigital matting and compositing
The lost world (1925) The lost world (1997)
Miniature, stop-motion Computer-generated images
Digital matting and compostingDigital matting and composting
King Kong (1933) Jurassic Park III (2001)
Optical compositingBlue-screen matting, digital composition, Optical compositing digital composition,
digital matte painting
Smith Duff Catmull Porter
O d 1996Oscar award, 1996
Titanic
M tti d C itiMatting and Compositing
backgroundreplacement
background
M tti d C iti
editing
Matting and Compositing
Digital matting: bluescreen mattingDigital matting: bluescreen matting
Forrest Gump (1994)
• The most common approach for films.• Expensive, studio setup.p p• Not a simple one-step process.
Color difference method (Ultimatte)Color difference method (Ultimatte)C=F+B F
Blue-screenphotograph
Spill suppressionif B>G then B=G
Matte creation=B-max(G,R)p g p (G, )
demo with Paint Shop Pro (B=min(B,G))
Problems with color differenceProblems with color difference
Background color is usually not perfect! (lighting, shadowing…)
Chroma-keying (Primatte)Chroma keying (Primatte) Chroma-keying (Primatte)Chroma keying (Primatte)
demo
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
F BF B
foreground color alpha matte background plate
BFC )α(α 1CcompositeF BFC )α(α 1
compositingC
C iti
p gequationB =0
Compositing
F BF B
BFC )α(α 1CcompositeF BFC )α(α 1
compositingC
C iti
p gequationB
C=1
Compositing
F BF B
BFC )α(α 1CcompositeFC
BFC )α(α 1
compositing
C iti
p gequationB =0.6
Compositing
F BF B
CobservationBFC )α(α 1 BFC )α(α 1
compositing
M tti
p gequation
Matting
F BF B
CBFC )α(α 1
Three approaches:
1 d # k BFC )α(α 1
compositing
1 reduce #unknowns
2 add observations
M tti
p gequation3 add priors
Matting
F BBF BB
CBFC )α(α 1 BFC )α(α 1
difference
M tti ( d # k )
differencematting
Matting (reduce #unknowns)
F F
CBFC )α(α 1
BBFC )α(α 1
blue screen
M tti ( d # k )
blue screenmatting
Matting (reduce #unknowns)
F F
BFC )α(α 1BC
BFC )α(α 1BFC )α(α 1
M tti ( dd b ti )triangulation
Matting (add observations)
F BF B
BCBFC )α(α 1BG BFC )α(α 1
FGunknown
Natural image mattingM tti ( dd i )rotoscopingRuzon-Tomasi
unknown
Natural image mattingMatting (add priors)
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
para observedf(z)+z ypara-
metersobserved
signal
)|(max* yzPz Example: super-resolution)|(max yzPz
z)()|(max zPzyP
super-resolutionde-blurringde-blocking
)(max
yPz de-blocking
…
B i f k
)()|(max zLzyLz
Bayesian framework
para observedf(z)+z ypara-
metersobserved
signal
)()|(max* zLzyLz )()|(max zLzyLzz
2)(zfy data a-priori
B i f k
2evidence knowledge
Bayesian framework
posterior probability
likelihood priors
B i f kBayesian framework
P iPriors B i ttiBayesian matting
repeat1. fix alpha
2. fix F and B
O ti i tiuntil converge
Optimization
DDemo
inputtrimapalphainputtrimapalpha
R ltResults
input compositeinput composite
R ltResults
input imagetrimap
C iComparisons
Bayesian Ruzon-Tomasi
C iComparisons
Bayesian Ruzon-Tomasi
C iComparisons
MishimaMishima
C iComparisons
BayesianBayesian
C iComparisons
input image
C iComparisons
Bayesian Mishima
C iComparisons
Bayesian Mishima
C iComparisons
i tinputvideo
Vid ttiVideo matting
i tinputvideo
inputkeytrimaps
Vid ttiVideo matting
i tinputvideo
interpo-latedtrimaps
Vid ttiVideo matting
inp tinputvideo
interpo-t tlated
trimapsoutputalpha
Vid ttiVideo matting
Compo-inp t Compo-site
inputvideo
interpo-t tlated
trimapsoutputalpha
Vid ttiVideo matting
optical flow
optical flow
S l itSample composite
G b ttGarbage mattes G b ttGarbage mattes
B k d ti tiBackground estimation B k d ti tiBackground estimation
Al h ttAlpha matte
without b k d
withb k d
C i
background background
Comparison
C
P(F)
B
Problems with Bayesian mattingProblems with Bayesian matting
• It requires fine trimaps for good results• It requires fine trimaps for good results• It is tedious to generate fine trimaps• Its performance rapidly degrades when
foreground and background patterns foreground and background patterns become complex Th i di t d l l t l t th • There is no direct and local control to the resulting mattes
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
Scribble based inputScribble-based input
trimap scribblep
MotivationMotivation
LazySnappingLazySnapping
L S in-th mean foreground color
LazySnapping
L S iLazySnapping L S iLazySnapping
Matting approachesMatting approaches• Sampling approaches: solve for each • Sampling approaches: solve for each
alpha separately by utilizing local fg/bg samples, e.g. Ruzon/Tomasi, Knockout and Bayesian mattingKnockout and Bayesian matting.
• Propagation approaches: solve the whole matte together by optimizing, e g Poisson BP random walker e.g. Poisson, BP, random walker, closed-form and robust matting.
Poisson mattingPoisson matting
Poisson mattingPoisson matting Robust mattingRobust matting• Jue Wang and Michael Cohen CVPR • Jue Wang and Michael Cohen, CVPR
2007
Robust mattingRobust matting• Instead of fitting models a non-• Instead of fitting models, a non-
parametric approach is used
B i R bBayesian Robust
Robust mattingRobust matting• We must evaluate hypothesized • We must evaluate hypothesized
foreground/background pairs
Bj C
Fidistance ratio
F
Robust mattingRobust matting• To encourage pure fg/bg pixels add • To encourage pure fg/bg pixels, add
weights
B F1C
FF2
Robust mattingRobust matting• Combine them together Pick up the • Combine them together. Pick up the
best 3 pairs and average themconfidence
Robust mattingRobust matting Robust mattingRobust matting
matte confidencematte confidence
Matte optimizationMatte optimization
Solved by Random Walk Algorithm
Matte optimizationMatte optimizationdata constraintsdata constraints
neighborhood constraints
Demo (EZ Mask)Demo (EZ Mask) EvaluationEvaluation• 8 images collected in 3 different • 8 images collected in 3 different
ways• Each has a “ground truth” matte
EvaluationEvaluation• Mean square error is used as the • Mean square error is used as the
accuracy metric• Try 8 trimaps with different accuracy
for testing robustness for testing robustness • 7 methods are tested: Bayesian, y ,
Belief propagation, Poisson, Random Walk KnockOut2 Closed Form and Walk, KnockOut2, Closed-Form and Robust mattingg
Quantitative evaluationQuantitative evaluation Subjective evaluationSubjective evaluation
Subjective evaluationSubjective evaluation Ranks of these algorithmsRanks of these algorithms
Poissonaccuracy
6 9robustness
6 8PoissonRandom walk
6.96.0
6.84.4
Knockout2Bayesian
4.53 9
4.56 0Bayesian
Belief PropagationClose form
3.93.32 6
6.03.12 0Close-form
Robust matting2.61.0
2.01.3
SummarySummary• Propagation-based methods are more • Propagation-based methods are more
robust• Sampling-based methods often
generate more accurate mattes than generate more accurate mattes than propagation-based ones with fine trimaps Robust matting combines strengths • Robust matting combines strengths of both
New evaluation (CVPR 2009)New evaluation (CVPR 2009)• http://www alphamatting com/• http://www.alphamatting.com/
Soft scissorSoft scissor• Jue Wang et al SIGGRAPH 2007• Jue Wang et. al., SIGGRAPH 2007• Users interact in a similar way to y
intelligent scissors
FlowchartFlowchart
FlowchartFlowchart Soft scissorSoft scissor
Demo (Power Mask)Demo (Power Mask) OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
Matting with multiple observationsMatting with multiple observations• Invisible lightsg
– Polarized lightsInfrared– Infrared
• Thermo-key• Depth Keying (ZCam)• Flash matting• Flash matting
I i ibl li ht (I f d)Invisible lights (Infared)
I i ibl li ht (I f d)Invisible lights (Infared) I i ibl li ht (I f d)Invisible lights (Infared)
I i ibl li ht (I f d)Invisible lights (Infared) I i ibl li ht (I f d)Invisible lights (Infared)
I i ibl li ht (I f d)Invisible lights (Infared) I i ibl li ht (P l i d)Invisible lights (Polarized)
I i ibl li ht (P l i d)Invisible lights (Polarized) Th KThermo-Key
Th KThermo-Key ZCZCam
D f ttiDefocus mattingvideo
M tti ith video
Matting with camera arrays
flash no flash matteflash no flash matte
Fl h ttiFlash matting
Background is much further than foreground and receives almost no flash light
Fl h ttiFlash matting
Foreground flash matting equationg g q
Generate a trimap and directly apply Bayesian matting.
Fl h ttiFlash matting
F d fl h ttiForeground flash matting J i t B i fl h ttiJoint Bayesian flash matting
J i t B i fl h ttiJoint Bayesian flash matting
flash no flash
C iComparison
foreground flash matting
ioint Bayesian flash matting
C iComparison Fl h ttiFlash matting
OutlineOutline• Traditional matting and compositing• Traditional matting and compositing• The matting problemg p• Bayesian matting and extensions• Matting with less user inputs
Matting with multiple observations• Matting with multiple observations• Beyond the compositing equation*Beyond the compositing equation• Conclusions
ConclusionsConclusions• Matting algorithms improves a lot in • Matting algorithms improves a lot in
these 10 years d ll l • In production, it is still always
preferable to shoot against uniform p gbackgroundsAlgorithms for more complex • Algorithms for more complex backgrounds
• Devices or algorithms for automatic mattingmatting
Thanks for your attention!Thanks for your attention! Shadow mattingand composting
target backgroundsource scene target backgroundblue screen image
target backgroundblue screen composite photographblue screen composite
photographblue screen composite
G t i Geometric errors
photographblue screen composite
Ph t t i Photometric errors
S LS L
C
S LS L
C=L+(1-)Sshadowcompositing
Cp g
equation
Sh d iti tiShadow compositing equation
C=L+(1-)Sshadowcompositing
Cp g
equation
Sh d ttiShadow matting
S LS L
C=L+(1-)Sshadowcompositing
Cp g
equation
Sh d ttiShadow matting
S LS L
C=L+(1-)Sshadowcompositing
Cp g
equation
Sh d ttiShadow matting
S LS L
C=L+(1-)Sshadowcompositing
Cp g
equation
Sh d itiShadow compositing
G t i Geometric errors
target backgroundsource scene
target backgroundsource scene
R i t #1Requirement #1
target backgroundsource scene
R i t #2Requirement #2
E i t ttiEnvironment matting
photographblue screen matting
traditional compositing equation
BFC T
BFC T
backgroundforegroundcomposite
g
environment compositing equation [Zongker’99]
RA
BFC T
BFC T
backgroundforegroundcomposite
g
O(k) images
E i t tti [Z k ’99]Environment matting [Zongker’99]
photographZongker et al.
P bl l di iProblem: color dispersion
photographZongker et al.
P bl l fProblem: glossy surface
photographZongker et al.
P bl lti l iProblem: multiple mappings
WW
C TC Tweighting backgroundweightingfunction
gcomposite
A bit i hti f tiM lti d l i t d G iArbitrary weighting functionMultimodal oriented Gaussian
high accuracy photographZongker et al.high accuracyalgorithm
P bl l di iProblem: color dispersion
high accuracy photographZongker et al.high accuracyalgorithm
Gl fGlossy surface
withoutwithoutwith photographwithoutorientation
withoutorientation
withorientation
withoutorientationorientation
O i t d G iOriented Gaussian
high accuracy photographZongker et al.high accuracyalgorithm
P bl lti l iProblem: multiple mappings
WR
A
BFC T
BFC T
backgroundweightingforegroundcomposite
bac g ou dweightingfunction
3 3 1 433
3 observations
3 1 43
3 observations11 variables
A RA, R F
433 433
3 observations7 variables3 observations
A RA, R F
413 413
3 observations5 variables3 observations
A R A A, R
A, colorless
F
T(c c )13T(cx , cy)
1113
3 observations
11
3 variables3 observations
A R A cx cy A, R
A, colorless
cx , cy , specularly
F refractive
Stimulus functionStimulus functionx
Stimulus functionStimulus functionx
yA
y
T( )M(T , A) T(cx, cy)
(cx, cy)
T
Ideal plane in RGB cubeIdeal plane in RGB cubeIdeal plane in RGB cubeIdeal plane in RGB cube
x
y
T(x y)T(x,y)
Calibrated manifold in RGB cubeCalibrated manifold in RGB cubeCalibrated manifold in RGB cubeCalibrated manifold in RGB cube
x (cx ,cy)C (cx ,cy)C
T(x y)y
T(x,y)
Estimate c c and Estimate c c and
W
Estimate cx ,cy and Estimate cx ,cy and
backgroundif ld
W
Tmanifold
C (cx ,cy)
C = KC
C
KC
KK
Problem: noisy matteProblem: noisy matte Edge-preserving filteringEdge-preserving filteringEdge preserving filteringEdge preserving filtering
with filteringwithout filtering gg
Input imageInput image Difference thresholdingDifference thresholding
Morphological operationMorphological operation FeatheringFeathering
Heuristics for specular highlightsHeuristics for specular highlights
Wbackgroundmanifold
W
Tmanifold
CC >1+ C >1+
K
Heuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlights
cx cy
C=T(c c )C=T(cx, cy)
Heuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlightsHeuristics for specular highlights
==
input estimation foregroundinput estimation foreground(highlights)
Composite with highlightsComposite with highlights
mattingmethod
compositingmodel methodmodel
color blue-screenBFC )α(α 1
color blending
blue-screenBayesian
shadowShadowmattingβ)L(βSC 1
refraction
matting
High accuracyrefractionreflection
High-accuracyenv. matting WBFC