primal-dual coding to probe light transport
DESCRIPTION
Primal-Dual Coding to Probe Light Transport Matthew O'Toole, Ramesh Raskar, and Kiriakos N. Kutulakos. ACM SIGGRAPH, 2012. Abstract: We present primal-dual coding, a photography technique that enables direct fine-grain control over which light paths contribute to a photo. We achieve this by projecting a sequence of patterns onto the scene while the sensor is exposed to light. At the same time, a second sequence of patterns, derived from the first and applied in lockstep, modulates the light received at individual sensor pixels. We show that photography in this regime is equivalent to a matrix probing operation in which the elements of the scene's transport matrix are individually re-scaled and then mapped to the photo. This makes it possible to directly acquire photos in which specific light transport paths have been blocked, attenuated or enhanced. We show captured photos for several scenes with challenging light transport effects, including specular inter-reflections, caustics, diffuse inter-reflections and volumetric scattering. A key feature of primal-dual coding is that it operates almost exclusively in the optical domain: our results consist of directly-acquired, unprocessed RAW photos or differences between them.TRANSCRIPT
![Page 1: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/1.jpg)
Primal-Dual Codingto Probe Light Transport
Matthew O’TooleUniversity of Toronto
Ramesh RaskarMIT Media Lab
Kyros KutulakosUniversity of Toronto
![Page 2: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/2.jpg)
conventional photography
photo records all light paths that reach camera sensor
![Page 3: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/3.jpg)
primal-dual coding photography
photo only records a user-defined subset of light paths
![Page 4: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/4.jpg)
conventional photography
primal-dual coding photography
![Page 5: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/5.jpg)
new photography regime: primal-dual codingmodulate contribution of specific light path
contributions
transport probing equationgeneralizes conventional photography & confocal imaging [Wilson et al. 96]
combine illumination coding [Schechner et al. 07] & exposure coding [Hitomi et al. 11]
efficient primal-dual codes by stochastic matrix probingprovides convergence & optimality guarantees [Bekas et al. 07]
![Page 6: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/6.jpg)
1. the result of light transport from sources to sensor pixels
2. the image recorded by the camera’s sensor
what is a photo?
![Page 7: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/7.jpg)
transport matrix [Ng et al. 03]photo
light
conventional photography
degrees of freedom
![Page 8: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/8.jpg)
primal-dual coding photography
“probing matrix”
degrees of freedom
![Page 9: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/9.jpg)
multiply
primal-dual coding photography
“probing matrix”
degrees of freedom
![Page 10: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/10.jpg)
primal-dual coding photography
“probing matrix”
degrees of freedom
![Page 11: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/11.jpg)
primal-dual coding photography
degrees of freedom
![Page 12: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/12.jpg)
mathematical formulation of primal-dual coding photography
![Page 13: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/13.jpg)
![Page 14: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/14.jpg)
![Page 15: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/15.jpg)
![Page 16: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/16.jpg)
one-shot primal-dual coding photography
![Page 17: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/17.jpg)
experimental setup
SLR camera LCD mask
projector
scene
beamsplitter
![Page 18: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/18.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 19: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/19.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 20: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/20.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 21: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/21.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 22: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/22.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 23: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/23.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 24: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/24.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 25: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/25.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 26: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/26.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 27: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/27.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 28: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/28.jpg)
step 2illuminate scene
with vector . (primal code)
step 3attenuate image with vector .
(dual code)
step 4repeat times
step 1open shutter
step 5close shutter
![Page 29: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/29.jpg)
![Page 30: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/30.jpg)
Rademacher primal-dual codes
stochastic diagonal estimator [Bekas et al. 07]
primal codes are Rademacher random vectors: random vector in
dual codes derive from primal code:
codes converge to identity probing matrix:
variance of pixel for primal-dual codes
aperture correlation (microscopy) is a diagonal estimator [Wilson et al. 96, Levoy et al. 04]
stochastic estimator for general probingdual codes for general probing matrix :
![Page 31: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/31.jpg)
designing probing matrices
![Page 32: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/32.jpg)
designing probing matrices
transport matrix
![Page 33: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/33.jpg)
designing probing matrices
transport matrix
![Page 34: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/34.jpg)
designing probing matrices
camera
pro
jector
transport matrix
![Page 35: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/35.jpg)
designing probing matrices
camera
pro
jector
camera
pro
jector
transport matrix
![Page 36: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/36.jpg)
designing probing matrices
camera
pro
jector
camera
pro
jector
transport matrix
![Page 37: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/37.jpg)
designing probing matrices
camera
pro
jector
camera
pro
jector
transport matrix
![Page 38: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/38.jpg)
coaxial example: contrast-enhancing direct light
conventionalphoto
all light paths
![Page 39: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/39.jpg)
coaxial example: contrast-enhancing direct light
direct +½ indirect
conventionalphoto
direct + 1/2 indirect light paths
![Page 40: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/40.jpg)
coaxial example: contrast-enhancing direct light
direct +½ indirect
conventionalphoto
direct +¼ indirect
direct +1/16 indirect
direct + 1/16 indirect light paths
![Page 41: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/41.jpg)
coaxial example: capturing short to long range paths
conventional
all light paths
![Page 42: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/42.jpg)
coaxial example: capturing short to long range paths
conventional
indirect
indirect light paths
![Page 43: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/43.jpg)
coaxial example: capturing short to long range paths
conventional
indirect mid-range indirect
medium to long range indirect light paths
![Page 44: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/44.jpg)
coaxial example: capturing short to long range paths
conventional
indirect mid-range indirect
long-range indirect
long range indirect light paths
![Page 45: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/45.jpg)
coaxial example: separating light transport effects
conventional
all light paths
![Page 46: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/46.jpg)
coaxial example: separating light transport effects
conventional
indirect
indirect light paths
![Page 47: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/47.jpg)
coaxial example: separating light transport effects
conventional
indirect direct + backscatter
direct + back-scatter light paths
![Page 48: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/48.jpg)
coaxial example: separating light transport effects
indirect [Nayar et al. 06]
indirect direct + backscatter
direct [Nayar et al. 06]
![Page 49: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/49.jpg)
coaxial example: separating light transport effects
indirect direct + backscatter
low-freq. indirect high-freq. indirect
![Page 50: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/50.jpg)
conventional
all light paths
coaxial example: de-scattering [Fuchs et al. 08]
![Page 51: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/51.jpg)
conventional de-scattered image
de-scattered diagonal
coaxial example: de-scattering [Fuchs et al. 08]
![Page 52: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/52.jpg)
coaxial example: de-scattering [Fuchs et al. 08]
conventional
direct + 1/16 indirect
direct + 1/16 indirect light paths
de-scattered image
![Page 53: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/53.jpg)
conventional
direct + 1/16 indirect shifted indirect
de-scattered image
coaxial example: de-scattering [Fuchs et al. 08]
shifted indirect + 1/16 light paths
![Page 54: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/54.jpg)
non-coaxial example: making 3D regions invisible
all light paths
![Page 55: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/55.jpg)
“privacy zone”
non-coaxial example: making 3D regions invisible
![Page 56: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/56.jpg)
non-coaxial example: making 3D regions invisible
![Page 57: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/57.jpg)
non-coaxial example: making 3D regions invisible
![Page 58: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/58.jpg)
non-coaxial example: making 3D regions invisible
![Page 59: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/59.jpg)
privacy zone
non-coaxial example: making 3D regions invisible
![Page 60: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/60.jpg)
non-coaxial example: color coding regions of space
color coded zones
![Page 61: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/61.jpg)
non-coaxial example: color coding regions of space
![Page 62: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/62.jpg)
non-coaxial example: color coding regions of space
![Page 63: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/63.jpg)
real-time PDC photographymatching 1440Hz DLP kits for illumination and attenuation
![Page 64: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/64.jpg)
real-time PDC photography
projector
matching 1440Hz DLP kits for illumination and attenuation
![Page 65: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/65.jpg)
real-time PDC photography
mask
camera
matching 1440Hz DLP kits for illumination and attenuation
DMD
![Page 66: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/66.jpg)
![Page 67: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/67.jpg)
additional details on primal-dual coding
non-negative primal-dual codesmany cases realized with strictly non-negative codesfor all others, subtract two photos computationally
codes robust to optical misalignmentsaccount for both spatial and temporal misalignments
one-shot versus multi-shotone-shot limited by projector display ratemulti-shot limited by camera read noise
![Page 68: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/68.jpg)
concluding remarks
primal-dual coding photographyproduces new types of photos governed by probing equation
combines coded illumination and coded exposure
guarantees optimality and convergence rate of codes
can be a see-through device
future worknew possibilities for optical applications
combine with existing computational photography techniques (image-based relighting,geometry acquisition, …)
![Page 69: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/69.jpg)
Matthew O’TooleUniversity of Toronto
Ramesh RaskarMIT Media Lab
Kyros KutulakosUniversity of Toronto
Primal-Dual Codingto Probe Light Transporthttp://www.dgp.toronto.edu/~motoole/primaldualcoding.html
![Page 70: Primal-Dual Coding to Probe Light Transport](https://reader033.vdocuments.net/reader033/viewer/2022060200/559838e91a28ab41628b46b7/html5/thumbnails/70.jpg)
64 codes
16 codes 32 codes8 codes
128 codes 256 codes