![Page 1: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/1.jpg)
Poisson Ma)ng Jian Sun, Jiaya Jia, Chi-‐Keung Tang, heung-‐Yeung Shum
Presenter: Vera Dadok Discussant: Eileen Bai
![Page 2: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/2.jpg)
Recall: Ma)ng • Ma)ng equaEon over image
– F – foreground – B – background – α – alpha maKe
ResulEng alpha maKe Image I Source: hKp://www.cse.cuhk.edu.hk/~leojia/all_project_webpages/Poisson%20ma)ng/poisson_ma)ng.html
BFI )1( αα −+=
![Page 3: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/3.jpg)
Recall: Ma)ng • Ma)ng equaEon over image
• Under-‐constrained equaEon: – 7 unknown variables
• α • R,G,B channels of F • R,G,B channels of B
– 3 known variables • R,G,B channels of I
• AddiEonal informaEon – User-‐defined Trimap
Trimap
Image I
BFI )1( αα −+=
![Page 4: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/4.jpg)
Poisson Ma)ng Another approach to solving the ma)ng problem Approach:
– Use gradient of image to calculate gradient of maKe – Reconstruct maKe from gradient of maKe
Key addi.on: – Enables user interacEon to locally improve results
![Page 5: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/5.jpg)
Poisson Ma)ng • Ma)ng equaEon
• Gradient of ma)ng equaEon
• Gradient rewriKen
• Once gradient of α is obtained, we can create maKe
BFBFI ∇−+∇+∇−=∇ )1()( ααα
BFI )1( αα −+=
A = 1F −B
D =α∇F − (1−α)∇B))1((1 BFI
BF∇−−∇−∇
−=∇ ααα
![Page 6: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/6.jpg)
Poisson Ma)ng • Ma)ng equaEon
• Gradient of ma)ng equaEon
• Gradient rewriKen
• Once gradient of α is obtained, we can create maKe
)( DIA −∇=∇α
BFBFI ∇−+∇+∇−=∇ )1()( ααα
BFI )1( αα −+=
A = 1F −B
D =α∇F − (1−α)∇B
![Page 7: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/7.jpg)
Poisson Ma)ng: Two steps
• Step 1: Global (required) – Solve global Poisson ma)ng problem, assumes
• Step 2: Local (opEonal) – Adjust local regions with local Poisson ma)ng operaEons
(1) Global Poisson Ma)ng (2) Local Poisson Ma)ng
)( DIA −∇=∇α
D = 0
![Page 8: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/8.jpg)
Step 1: Global Poisson Ma)ng
Global soluEon assumes D = 0 in gradient equaEon
– i.e. assumes “smooth” foreground and background
IntuiEvely – MaKe gradient is approximately proporEonal to image gradient
)( DIA −∇=∇α
A = 1F −B
D =α∇F − (1−α)∇B
∇α ≈∇IF −B
D =α∇F − (1−α)∇B = 0
General Case Global Case with AssumpEon
![Page 9: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/9.jpg)
What does ‘smooth’ mean? D =α∇F − (1−α)∇B = 0
BFBFI ∇−+∇+∇−=∇ )1()( ααα
Color Image Grayscale I
Solve for alpha using smooth assump.on, D=0 What should the ma>e edge be?
![Page 10: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/10.jpg)
What does ‘smooth’ mean? D =α∇F − (1−α)∇B = 0
BFBFI ∇−+∇+∇−=∇ )1()( ααα
Grayscale I
Gradient α (in x) Gradient I (in x)
α
∇α ≈∇IF −B
![Page 11: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/11.jpg)
What does ‘smooth’ mean? D =α∇F − (1−α)∇B = 0
BFBFI ∇−+∇+∇−=∇ )1()( ααα
F
α
B Does this seem like a good ma>e? Was this “smooth”?
![Page 12: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/12.jpg)
What does ‘smooth’ mean?
Color Image Grayscale I
D =α∇F − (1−α)∇B = 0BFBFI ∇−+∇+∇−=∇ )1()( ααα
What should the ma>e edge be?
![Page 13: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/13.jpg)
What does ‘smooth’ mean?
Color Image
Does this seem like a good ma>e? Was this “smooth” assump.on good?
F
α
B
D =α∇F − (1−α)∇B = 0BFBFI ∇−+∇+∇−=∇ )1()( ααα
![Page 14: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/14.jpg)
Step 1: Global SoluEon method This simplified equaEon is solved in grayscale channel
– 2 known variables: • gradients of I
– 3 unknown variables: • (F-‐B), treated as a single variable • gradients of α
∇α ≈∇IF −B
+ AddiEonal knowledge • Boundary condiEons from Trimap
![Page 15: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/15.jpg)
Step 1: Global SoluEon method Minimize error in
In the sense of least squares
This has an associated Poisson equaEon with Dirichlet boundary condiEons (due to Trimap), which we know how to solve
∇α ≈∇IF −B
This is solved iteraEvely, approximaEng F-‐B, then solving for α, then repeat
![Page 16: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/16.jpg)
Global Poisson Ma)ng
Results good for smooth F and B (thus D≈0)
Source: hKp://www.cse.cuhk.edu.hk/~leojia/all_project_webpages/Poisson%20ma)ng/poisson_ma)ng.html
![Page 17: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/17.jpg)
Why do we need local operaEons?
Global Poisson maFng assumed D=0
Exact gradient equaEon
)( DIA −∇=∇αA = 1
F −BD =α∇F − (1−α)∇B
![Page 18: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/18.jpg)
Step 2: Local OperaEons
Do not assume D=0 • Goal: change A, D to produce beKer approximaEon of α
– ‘Fix’ problematic areas in global result • Two categories
– Channel SelecEon – Local filters (brushes)
Δα = div(A(∇I −D))
)( DIA −∇=∇α
• Aner local α-‐gradient changes solve local Poisson equaEon with Dirichlet condiEons – Key: local changes in α-‐gradient do not produce
arEfacts in final maKe
![Page 19: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/19.jpg)
Local OperaEons: Channel SelecEon • Create new weighted channel of R, G, B to minimize |D|
– Instead of grayscale channel – Reduces error in D
• Process – User selects background samples – AutomaEcally calculates channel weights to minimize variance of foreground and background colors
)( DIA −∇=∇α
grayscale
new weighted channel
α maKe channel
![Page 20: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/20.jpg)
Local OperaEons: High-‐pass filter
• Low pass filter on D • High-‐pass on α due to term – D is esEmated using low-‐frequencies of gradient
before
aner
)( DIA −∇=∇α
(∇I −D)
MaKe gradient MaKe
![Page 21: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/21.jpg)
Local OperaEons: BoosEng brush
• Increase or decrease sharpness
• Gaussian filter on A
before
aner
)( DIA −∇=∇α
MaKe gradient MaKe
![Page 22: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/22.jpg)
Local OperaEons: Diffusion filtering
• Remove JPEG arEfacts
before
aner
)( DIA −∇=∇α
MaKe gradient MaKe
![Page 23: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/23.jpg)
Local OperaEons: Clone Brush
• Exactly copy gradient )( DIA −∇=∇α
MaKe gradient MaKe
Erase brush Inverse brush
![Page 24: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/24.jpg)
Full algorithm summary 0: Create Trimap (user defined) 1: Global Poisson ma)ng (automaEc) 2: Apply local Poisson operators (user defined)
(i) Apply channel selecEon (smoother background or foreground, reduces |D|) (ii) Apply highpass filtering to obtain approximaEon of D (iii) Apply boosEng brush to change A (iv) Apply clone brush to disEnguish gradients
Note: the order of these steps is important! Erase brush and inverse brush can be applied any Eme
![Page 25: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/25.jpg)
Video break
Global SoluEon: 1:10 Local SoluEons: 1:30
![Page 26: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/26.jpg)
Full algorithm summary
Source: hKp://www.cse.cuhk.edu.hk/~leojia/all_project_webpages/Poisson%20ma)ng/poisson_ma)ng.html
User (Trimap)
![Page 27: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/27.jpg)
Comparison to other methods
• Via picture
![Page 28: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/28.jpg)
Comparison to other methods
• Via picture
![Page 29: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/29.jpg)
Improvements over other methods
• Natural image ma)ng including Bayesian Ma)ng – Not easy to adjust output of algorithms if mistakes are made in alpha maKe
• Difference ma)ng and triangulaEon ma)ng – Need mulEple input images
![Page 30: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/30.jpg)
ApplicaEons
• Image composiEng
![Page 31: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/31.jpg)
Other applicaEons
• De-‐fogging
![Page 32: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/32.jpg)
Other ApplicaEons
• MulE-‐background
![Page 33: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/33.jpg)
Thank you!
QuesEons?
![Page 34: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/34.jpg)
Discussion
![Page 35: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/35.jpg)
LimitaEons 1.) User-‐defined local operaEons
l Relies on the user for input l Good enough? l Background knowledge of brushes
2.) Specific order of operaEons l Restricts freedom/flexibility
3.) Similar foreground/background elements
l Hard to disEnguish underlying structure of maKe l Difficult for computers as well as humans
*First two points could be argued as posiEve qualiEes as well.
4.) MulEple disEnct foreground elements l Requires complicated trimap l Bayesian could solve relaEvely simply
![Page 36: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/36.jpg)
Improvements
l Ideas? l Separate “hair” from “fur” - These two come up the most onen - Have very different properEes
![Page 37: Poisson&Mang& - Visualization Lab | UC Berkeleyvis.berkeley.edu/courses/cs294-69-fa11/wiki/images/5/5f/Poisson... · Poisson&Mang& JianSun,& Jiaya&Jia,&Chi2Keung&Tang,&heungYeung](https://reader034.vdocuments.net/reader034/viewer/2022051507/5a730ebf7f8b9aa2538e51de/html5/thumbnails/37.jpg)
Future Work
l Extends naturally to... Video ma)ng! - “Real-‐Time Video Ma)ng Using MulEchannel Poisson EquaEons” -‐ 2010
- AutomaEcally generated trimaps
l CombinaEon of Poisson and Bayesian Ma)ng - Combine user-‐defined local operaEons with the pixel by pixel
esEmaEon in Bayes'