extracting and manipulating lighting through intrinsic image...
TRANSCRIPT
![Page 1: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/1.jpg)
Extracting and manipulating lighting
through intrinsic image decomposition
Pierre-Yves Laffont
IMI Research Seminar
November 17, 2015
Slides shared for scientific dissemination
![Page 2: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/2.jpg)
Postdoctoral researcher at ETH Zürich (Switzerland)
with Markus Gross, since March 2014
Visiting the BeingThere Centre at NTU Singapore
since June 2014
2
![Page 3: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/3.jpg)
Postdoctoral researcher at Brown University (USA)
• with James Hays, 2012 – 2014 (16 months)
PhD from INRIA Sophia-Antipolis (France)
• with George Drettakis & Adrien Bousseau, 2009 – 2012
• visiting student researcher at MIT and UC Berkeley (USA)
3
![Page 4: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/4.jpg)
The appearance of a scene changes with many factors:
Viewpoint
Lighting
Time of day
Season
Weather
Material/geometry
changes (e.g. aging)
4
CRACOW 4 seasons timelapse
by Piotr Wancerz - www.timelapsemedia.pl
![Page 5: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/5.jpg)
5
Exploring and Editing Scene Appearance
![Page 6: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/6.jpg)
What can those appearance variations
teach us about the scene?
6
image analysis / scene understanding
Exploring and Editing Scene Appearance
![Page 7: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/7.jpg)
What can those appearance variations
teach us about the scene?
Can we manipulate scene appearance
and synthesize new images?
7
image analysis / scene understanding
image editing / example-based transfer
Exploring and Editing Scene Appearance
![Page 8: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/8.jpg)
Understand and editoutdoor scene appearance with transient attributesSIGGRAPH 2014
Synthesize
novel views
EGSR 2011
Transfer
weathering
effects
TOG 2011
Extract and manipulate lightingwith intrinsic images
SIGGRAPH Asia 2012TVCG 2013TOG 2015ICCV 2015
![Page 9: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/9.jpg)
Understand and editoutdoor scene appearance with transient attributesSIGGRAPH 2014
Synthesize
novel views
EGSR 2011
Transfer
weathering
effects
TOG 2011
Extract and manipulate lightingwith intrinsic images
SIGGRAPH Asia 2012TVCG 2013TOG 2015ICCV 2015
![Page 10: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/10.jpg)
Motivation: Lighting matters
10
© Tim Smalley
![Page 11: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/11.jpg)
Motivation: Relighting for telepresence
11
Life-size 3D transparent displays in Zurich and Singapore
![Page 12: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/12.jpg)
Motivation: Relighting for telepresence
12
Participant and his
virtual representation
[Maimone13]
![Page 13: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/13.jpg)
Extracting and manipulating lighting
through intrinsic image decomposition
Slides shared for scientific dissemination
![Page 14: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/14.jpg)
Input image = Reflectance x Illumination
Intrinsic images
Decompose input image [Barrow-Tenenbaum78]
into reflectance and illumination layers
15
![Page 15: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/15.jpg)
Input image = Reflectance x Illumination
Intrinsic images
Decompose input image [Barrow-Tenenbaum78]
into reflectance and illumination layers
16
Per-pixel per-channel multiplication
![Page 16: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/16.jpg)
Decomposition example
17
Reflectance
Illumination
Input
[Laffont13]
![Page 17: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/17.jpg)
Ill-posed problem
need to constrain the decomposition
Intrinsic images
18
Reflectance3 unknowns
Radiance3 known values
Illumination3 unknowns
![Page 18: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/18.jpg)
Automatically from a single image
19
Analyze local variations [Land71, Horn86, Tappen05]
Incorporate global constraints [Shen08, Gehler11, Shen11]
Fine tune with crowdsourced database [Bell14]
![Page 19: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/19.jpg)
With user interaction
Propagate user scribbles [Bousseau09, Shen11]
Refine automatic results with scribbles [Bonneel14]
20
Input image User scribbles Reflectance Illumination
![Page 20: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/20.jpg)
From multiple views
Use extra information to constrain decomposition
Reconstruct sparse geometry
• Structure from motion
[Snavely06]
• Multi view stereo
[Furukawa09]
21
![Page 21: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/21.jpg)
From multiple views
From photocollections with varying lighting [Laffont12]
22
Input images Reflectance Illumination
![Page 22: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/22.jpg)
From multiple views
From photocollections with varying lighting [Laffont12]
23
Input image Re-lit image
![Page 23: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/23.jpg)
From multiple views
From multiple views with constant lighting [Laffont13]
24
Sunillumination
Skyillumination
Indirectillumination
Outdoor photographs
Reflectance
![Page 24: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/24.jpg)
From multiple views
From multiple views with constant lighting [Laffont13]
25
Edited reflectance Virtual object insertion Sunset relighting
![Page 25: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/25.jpg)
From multiple views
From multiple views with constant lighting [Laffont13]
Relighting with moving shadows [Duchêne15]
26
![Page 26: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/26.jpg)
From timelapse sequences
Fixed viewpoint, varying lighting [Weiss01, Matsushita04]
[Sunkavalli08, Hauagge13]
27
![Page 27: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/27.jpg)
Intrinsic decomposition of image sequences from local temporal variations
Pierre-Yves Laffont, Jean-Charles Bazin
ICCV 2015
= ×
![Page 28: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/28.jpg)
Input: Timelapse with N frames
Output: 1 reflectance image, N illumination images
Intrinsic images from a timelapse
29
Reflectance3 unknowns
Radiance3*N known values
Illumination3*N unknowns
![Page 29: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/29.jpg)
Intrinsic images from a timelapse
30
Reflectance Shading images
![Page 30: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/30.jpg)
Locally smooth illumination
In flat regions with no cast shadows,
the illumination is constant
31
![Page 31: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/31.jpg)
Locally smooth illumination
32
Locally adaptive
smoothing weights
![Page 32: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/32.jpg)
Results: Madrid sequence
33
Reflectance
Inp
ut
Illu
min
ation
![Page 33: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/33.jpg)
Results: City sequence
34
Reflectance
Inp
ut
Illu
min
ation
![Page 34: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/34.jpg)
Evaluation: St Basil synthetic benchmark
35
![Page 35: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/35.jpg)
Conclusion
Automatically decompose timelapse sequences
into reflectance and illumination sequences
Enforces smooth illumination locally
based on the observations
Synthetic benchmark for evaluation
https://graphics.ethz.ch/~plaffont/research/intrinsicTimelapse
36
![Page 36: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/36.jpg)
37
![Page 37: Extracting and manipulating lighting through intrinsic image decompositionimi.ntu.edu.sg/NewsEvents/Events/PastSeminars/Documents/17_Nov… · Extracting and manipulating lighting](https://reader035.vdocuments.net/reader035/viewer/2022062919/5ee178b3ad6a402d666c57ec/html5/thumbnails/37.jpg)
38
Laffont P.Y. and Bazin J.C. Intrinsic decomposition of image sequences from
local temporal variations. To be presented at IEEE International Conference on
Computer Vision (ICCV 2015), Santiago.
Duchene S., Riant C., Chaurasia G., Popov S., Laffont P.Y., Lopez-Moreno J.,
Bousseau A., and Drettakis G. Multi-view intrinsic images of outdoors scenes
with an application to relighting. ACM Transactions on Graphics, 2015. To be
presented at SIGGRAPH 2016, Anaheim.
Laffont P.Y., Bousseau A., Paris S., Durand F., and Drettakis G. Coherent
intrinsic images from photo collections. ACM Transactions on Graphics
(SIGGRAPH Asia), 2012. Presented at SIGGRAPH Asia 2012, Singapore.
Laffont P.Y., Bousseau A., and Drettakis G. Rich intrinsic image decomposition
of outdoor scenes from multiple views. IEEE Transactions on Visualization and
Computer Graphics, 2013. Selected as Spotlight Paper (Feb. 2013 issue).
Presented at SIGGRAPH 2012, Los Angeles (Poster and Talk sessions).