1. Optical Computing for Fast Light Transport Analysis Matthew OToole and Kyros Kutulakos University of Toronto

2. Forward Transport Inverse Transport Light Transport in a Real-World Scene photo of a scene illumination light transport defines how light interacts with a scene 3. The Light Transport Matrix [Debevec et al. 2000] photo pixels pattern pixels elements 4. optical domain Computing with Light Key idea: analyze the transport matrix by implementing iterative numerical algorithms directly in optics transport matrix illumination pattern photo numerical domain 5. Computing with Light transport matrix illumination pattern photo 1. project 2. capture numerical domain optical domain Key idea: analyze the transport matrix by implementing iterative numerical algorithms directly in optics 6. Computing with Light 1. project 2. capture numerical domain optical domain Key idea: analyze the transport matrix by implementing iterative numerical algorithms directly in optics 7. Computing with Light numerical domain optical domain Key idea: analyze the transport matrix by implementing iterative numerical algorithms directly in optics 8. Optical Computing for Fast Light Transport Analysis optical power iteration find principal eigenvector of optical Arnoldi find best rank-k approximation of optical GMRES given photo p, solve calibrated projectors calibrated cameras unknown, static scene scene physically accessible 9. project capture Eigenvector of a square matrix T when projected onto scene, we get the same photo back (multiplied by a scalar) Computing Transport Eigenvectors Numerical goal find such that and is maximalprojector camerabeam splitter 10. Optical Power Iteration Goal: find principal eigenvector of Observation: it is a fixed point of the sequence numerical domain Properties linear convergence [Trefethen and Bau 1997] eigenvalues must be distinct cannot be orthogonal to principal eigenvector 11. Optical Power Iteration numerical domain optical domain Goal: find principal eigenvector of Observation: it is a fixed point of the sequence 12. Optical Power Iteration optical domainnumerical domain projectcapture initialize normalize Goal: find principal eigenvector of Observation: it is a fixed point of the sequence 13. Optical Power Iteration optical domain projectcapture initialize normalize projector camera beam splitter Goal: find principal eigenvector of Observation: it is a fixed point of the sequence 14. Optical Power Iteration optical domain projectcapture initialize normalize projector camera beam splitter Goal: find principal eigenvector of Observation: it is a fixed point of the sequence 15. Optical Power Iteration optical domain projectcapture initialize normalize Goal: find principal eigenvector of Observation: it is a fixed point of the sequence 16. Optical Power Iteration optical domain projectcapture initialize normalize Goal: find principal eigenvector of Observation: it is a fixed point of the sequence 17. Optical Power Iteration optical domain projectcapture initialize normalize Goal: find principal eigenvector of Observation: it is a fixed point of the sequence 18. Optical Power Iteration optical domain projectcapture initialize normalize Goal: find principal eigenvector of Observation: it is a fixed point of the sequence 19. Optical Power Iteration optical domain (approximate) principal eigenvector Goal: find principal eigenvector of Observation: it is a fixed point of the sequence 20. Optical Power Iteration optical domain (approximate) principal eigenvector ... Goal: find principal eigenvector of Observation: it is a fixed point of the sequence 21. Computational microscopy [Bimber et al. 2010] connection to numerical methods not exploited Computational illumination [Wang et al. 2010] optical feedback avoided or ignored Krylov subspace methods [Saad 2003] use recurrence relations designed for extremely large matrices Computing with Feedback Loops Analog optical computing [Rajbenbach et al. 1987] encode matrix with optical masks optical feedback for iterative methods 22. Optical Computing for Fast Light Transport Analysis optical power iteration find principal eigenvector of optical Arnoldi find best rank-k approximation of optical GMRES given photo p, solve calibrated projectors calibrated cameras unknown, static scene scene physically accessible 23. Space of Camera-Projector Arrangements one viewpoint? yes coaxial optical paths? [Garg et al, 2006] point-source illumination? [Wang et al, 2009] 24. Space of Camera-Projector Arrangements one viewpoint? yes symmetric coaxial optical paths? [Garg et al, 2006] point-source illumination? [Wang et al, 2009] 25. Space of Camera-Projector Arrangements one viewpoint? no coaxial optical paths? [Garg et al, 2006] point-source illumination? [Wang et al, 2009] yes symmetric 26. Space of Camera-Projector Arrangements one viewpoint? yes no nonsymmetricsymmetric coaxial optical paths? [Garg et al, 2006] point-source illumination? [Wang et al, 2009] 27. Space of Camera-Projector Arrangements one viewpoint? yes no nonsymmetricsymmetric coaxial optical paths? [Garg et al, 2006] no point-source illumination? [Wang et al, 2009] yes 28. Space of Camera-Projector Arrangements one viewpoint? yes no nonsymmetricsymmetric coaxial optical paths? [Garg et al, 2006] yes no computable computable point-source illumination? [Wang et al, 2009] 29. Space of Camera-Projector Arrangements one viewpoint? yes no nonsymmetricsymmetric coaxial optical paths? [Garg et al, 2006] yes no computable computable point-source illumination? [Wang et al, 2009] yes no 30. one viewpoint? coaxial optical paths? [Garg et al, 2006] point-source illumination? [Wang et al, 2009] dense, low-rank sparse, high-rank no nonsymmetric no computable yes no yes symmetric yes computable Space of Camera-Projector Arrangements 31. Domain of Optical Arnoldi one viewpoint? yes no nonsymmetricsymmetric coaxial optical paths? [Garg et al, 2006] yes no computable computable point-source illumination? [Wang et al, 2009] yes no sparse, high-rankdense, low-rank 32. Domain of Optical GMRES one viewpoint? no nonsymmetric coaxial optical paths? [Garg et al, 2006] no computable point-source illumination? [Wang et al, 2009] symmetric computable dense, low-rank sparse, high-rank 33. Optical Computing for Fast Light Transport Analysis optical power iteration find principal eigenvector of optical Arnoldi find best rank-k approximation of optical GMRES given photo p, solve calibrated projectors calibrated cameras unknown, static scene scene physically accessible 34. Rank-k Transport Approximation Numerical goal [Simon and Zha 2000] find matrices such that Symmetric 1 camera, 1 projector 2 photos for rank- approx. Nonsymmetric 2 cameras, 2 projectors 4 photos for rank- approx. projector camerabeam splitter 35. Numerical goal [Simon and Zha 2000] find matrices that minimize Rank-k Transport Approximation Symmetric 1 camera, 1 projector 2 photos for rank- approx. Nonsymmetric 2 cameras, 2 projectors 4 photos for rank- approx. projector camerabeam splitter 36. Prior Work on Transport Acquisition Brute force [Schechner et al. 2007] [Sen et al. 2005] must capture to completion Methods draw from set of scene independent illumination Kernel Nystrm [Wang et al. 2009] designed for dense, low-rank matrices requires intense illumination and HDR photography Compressive sensing [Peers et al. 2009] [Sen and Darabi 2009] designed for sparse, high-rank matrices computationally very intensive 37. Optical Arnoldi for Symmetric T numerical domain Goal: compute matrices such that 38. Optical Arnoldi for Symmetric T numerical domain Goal: compute matrices such that 39. Optical Arnoldi for Symmetric T numerical domain Goal: compute matrices such that 40. Optical Arnoldi for Symmetric T numerical domain optical domain projectcapture initialize orthogonalize projected patterns captured photos Goal: compute matrices such that 41. Optical Arnoldi for Symmetric T numerical domain optical domain projectcapture initialize projected patterns captured photos orthogonalize Goal: compute matrices such that 42. projector camera beam splitter Optical Arnoldi for Symmetric T optical domain projectcapture initialize projected patterns captured photos orthogonalize Goal: compute matrices such that 43. Optical Arnoldi for Symmetric T optical domain projectcapture initialize projected patterns captured photos orthogonalize Goal: compute matrices such that rows of ... projected patterns 44. Optical Arnoldi for Symmetric T ...columns of Goal: compute matrices such that captured photos ... rows of projected patterns 45. optical domain projectcapture initialize projected patterns captured photos orthogonalize leftright left camera & projector right camera & projector Optical Arnoldi for Nonsymmetric T Goal: compute matrices such that 46. optical domain projectcapture initialize projected patterns captured photos normalize orthogonalize leftright captureproject leftright left camera & projector right camera & projector Optical Arnoldi for Nonsymmetric T Goal: compute matrices such that 47. left camera & projector Optical Arnoldi for Nonsymmetric T optical domain projectcapture initialize left-projected patterns right-captured photos normalize Goal: compute matrices such that right camera & projector orthogonalize leftright captureproject leftright 48. left camera & projector Optical Arnoldi: Implementation right camera & projector Just 1 line of MATLAB code: cameras & projectors calibrated RAW images, LDR capture Timings: 12s per Arnoldi iteration left-project & right-capture right-project & left-capture 49. Results: Optical Arnoldi 50. Results: Optical Arnoldi versus Nystrm 51. Results: Error Comparisons of Low-Rank Methods relativeerror 0 0 256 0.5 optimal (SVD) relative error , : ground truth, : rank- approx. # iterations 52. Results: Error Comparisons of Low-Rank Methods relativeerror 0 0 256 0.5 optimal (SVD) Nystrm relative error , : ground truth, : rank- approx. # iterations 53. Results: Error Comparisons of Low-Rank Methods relativeerror 0 0 256 0.5 optimal (SVD) Nystrm kernel Nystrm relative error , : ground truth, : rank- approx. # iterations 54. Results: Error Comparisons of Low-Rank Methods relativeerror 0 0 256 0.5 optimal (SVD) Nystrm kernel Nystrm optical Arnoldi relative error , : ground truth, : rank- approx. # iterations 55. Optical Computing for Fast Light Transport Analysis optical power iteration find principal eigenvector of optical Arnoldi find best rank-k approximation of optical GMRES given photo p, solve calibrated projectors calibrated cameras unknown, static scene scene physically accessible 56. Numerical goal given photo , find illumination that minimizes Light Transport Inversion Remarks low-rank or high-rank unknown & not acquired illumination sequence will be specific to input photo projector camerabeam splitter 57. Defocus Compensation [Zhang and Nayar 2006] inverts defocus kernel Prior Work on Transport Inversion Radiometric Compensation [Wetzstein and Bimber 2007] inversion after capturing matrix Transport matrix is known for all methods Interreflectance Cancellation [Seitz et al. 2005] decomposition into n-bounce images Stratified Inverse [Ng et al. 2009] [Bai et al. 2010] approximates inverse using geometric series 58. projector camera beam splitter optical domain projectcapture initialize projected patterns captured photos orthogonalize Optical GMRES for Symmetric T Goal: find illumination such that 59. Optical GMRES for Symmetric T optical domain projectcapture initialize orthogonalize Goal: find illumination such that input photo projector camera beam splitter projected patterns captured photos 60. Optical GMRES for Symmetric T optical domain projectcapture initialize orthogonalize Goal: find illumination such that input photo projector camera beam splitter 61. Results: Inversion for Low-Rank T 62. flashlight Results: Inversion for Low-Rank T 63. diffuser Results: Inversion for Low-Rank T 64. scene Results: Inversion for Low-Rank T 65. input photo Results: Inversion for Low-Rank T 66. input photo ? illumination Results: Inversion for Low-Rank T 67. Results: Inversion for Low-Rank T 68. relativeerror # iterations0 0 20 0.5convergence of photos actual (input) estimated Results: Inversion for Low-Rank T 69. relativeerror 0 0 20 0.5 relativeerror 0 0 20 1.0convergence of illumination actual (input) estimated actual estimated convergence of photos Results: Inversion for Low-Rank T # iterations # iterations 70. Results: Inversion for High-Rank T 71. lenses Results: Inversion for High-Rank T 72. illumination Results: Inversion for High-Rank T 73. Results: Inversion for High-Rank T 74. input photo input photo Results: Inversion for High-Rank T 75. ? input photo illumination Results: Inversion for High-Rank T 76. Results: Inversion for High-Rank T 77. Concluding Remarks Implement numerical algorithms directly in optics: methods that operate on large, unknown matrices are well established easy to build optical algorithms from many existing numerical implementations theoretical bounds for convergence rate Limitations: optical Arnoldi inefficient for high rank matrices optical GMRES only inverts a single image 78. Optical Computing for Fast Light Transport Analysis Matthew OToole and Kyros Kutulakos University of Toronto http://www.dgp.toronto.edu/~motoole