end-to-end learning for computational imaging - games-cn.org
Post on 21-Mar-2022
1 Views
Preview:
TRANSCRIPT
End-to-end Learning for Computational ImagingRealization of Diffractive Achromat and Super-resolution SPAD Camera
Xiong Dun
Tongji University
dunx@tongji.edu.cn
Institute of Precision Optical Engineering Key Laboratory of Advanced Micro-Structured Materials MOE
2020-01-09 1
Outline
Define and Background
Two examples
Conclusion
1. Learned achromatic DOE for full spectrum computational imaging
2. Optically Coded Super-resolution SPAD Camera
2
Computational Imaging
Solving an optimization problem:
decoupled
collaborative
integrated
Three ranks
3
Deep Learning
GEORGE BARBASTATHIS,2019
Based on neural network (NN). Composed by lots of simple nonlinear processing units, Each unit receives its inputs as weighted sums from the previous
4
Deep Learning based computational imagingDeep leaning used as a solver Deep leaning as a design framework
Rivenson et al. 2017
deblur
PR
Wu et al. 2018
Denoise
Chen et al. 2018
Lyu et al. 2017See
diffuser
Sitzmann et al. 2018
Chang et al. 2019 Metzler et al. 2019
5
Outline
Define and Background
Two examples
Conclusion
1. Learned achromatic DOE for full spectrum computational imaging
2. Optically Coded Super-resolution SPAD Camera
6
Problem statement
Diffractive Optical Elements (DOEs)
β’ Amplitude & Phase modulation
Thin; Great, flexible design space & operations
Wavelength sensitive β color dispersion
Metamerism problem
Peng et al. 2016
Need realize achromatic
design
7
Previous methods
Belong to collaborate. Design achromatic based on Handed-craft target PSF.
May not the optimal PSF for latter image processing
Peng et al. 2016
Meem et al. 2019 8
Proposed method
1D PSF simulator Solve the problem of Heavy memory consumption
οΏ½ππ
πΊπΊ(ππππ) πππππ½π½1 2ππππππππ β ππππβ1π½π½1 2ππππππππ
Origin rotational symmetric
Fresnel propagationπππ§π§β²,ππ = exp ππππ π₯π₯2 + π¦π¦2 + π§π§β²2 + ππ β 1 β π₯π₯,π¦π¦ β exp ππ
ππ2π§π§ π₯π₯2 + π¦π¦2
2
Discrete Hankel transform
For Nvidia 1080TI GPU, thelargest pixel sample numbers are2500 by 2500 with 3-wavelengthchannels
Past With proposed 1 D simulator
Can design pixel number of 8000by 8000 with 31-wavelengthchannels
πππ§π§β²,ππ =
10
Outline
Define and Background
Two examples
Conclusion
1. Learned achromatic DOE for full spectrum computational imaging
2. Optically Coded Super-resolution SPAD Camera
14
Problem statement
β’ Single photon level sensitivity
β’ Pico-second level time resolution
(Picture from MPD)
SPAD array sensor
β’ Low spatial resolution(32*64)
β’ Low fill-factor(3.15%)
Need realize super-resolution and anti-aliasing
(Picture by G.Intermite 2015)
15
Simulation result
Low fill-factor (no mask)
Low fill-factor
Full fill-factor
Full fill-factor
With phase mask
Ours
Ground Truth30.76 26.91 26.23
18
Real capture (the optimized mask is helpful)
Raw image without mask
Raw image with mask
Result without mask
Result with mask 19
75
0
25
50
Depth/mm
SPAD measurement summed over time axis Reconstruction depth map
Real capture (more results)
22
Outline
Define and Background
Two examples
Conclusion
1. Learned achromatic DOE for full spectrum computational imaging
2. Optically Coded Super-resolution SPAD Camera
24
Conclusion
1. Automatic seek the best-fit PSFs instead of non-optimized hand-crafted PSFs
2. Obtain good results even with a relatively simple image processingalgorithm
Example 1
Example 2
1. Achieve an spatial resolution enhancement of 4Γ2. Optimized compromise between sharpness and anti-aliasing for a
given pixel fill-factor.
Benefit from the end to end design framework
25
Conclusion
β’ Performance & robustness gainsβ’ Domain-specific hardware reduce footprint, cost, power,
capture timeβ¦
we envision the end to end design can boost lots of applications
26
Relative papers
1. Q.L. Sun, J. Zhang, X. Dun, B. Ghanem, Bernard, Y.F. Peng, Yifan and Heidrich, W.Heidrich, End-to-End Learned, Optically Coded Super-resolution SPAD Camera, ACMTransactions on Graphics, 2020
2. Y.F. Peng*, Q.L. Sun*, X. Dun*, G. Wetzstein, W. Heidrich, F. Heide. Learned LargeField-of-View Imaging With Thin-Plate Optics. ACM Transactions on Graphics, 2019.
3. X. Dun, Z.S. Wang, Y.F. Peng. Joint-designed achromatic diffractive optical element forfull-spectrum computational imaging (Invited Paper). Proc. of SPIE (2019).
4. D. S. Jeon, S.H. Baek, S. Yi, Q. Fu, X. Dun, W. Heidrich, M.H. Kim, Compact SnapshotHyperspectral Imaging with Diffracted Rotation, ACM Transactions on Graphics, (2019).
5. V. Sitzmann, S. Diamond, Y.F. Peng, X. Dun, S. Boyd, W. Heidrich, Felix Heide,Gordon Wetzstein. End-to-end Optimization of Optics and Image Processing forAchromatic Extended Depth of Field and Super-resolution Imaging. ACM Transactionson Graphics, 2018.
6. Q.L. Sun, X. Dun, Y.F. Peng, W. Heidrich, Depth and Transient Imaging withCompressive SPAD Array Cameras, in CVPR. 2018
27
Y.F. Peng
Stanford University
Q.L. Sun
KAUST
V. Sitzmann
Stanford University
M.H. Kim
KAIST
G. Wetzstein
Stanford University
W. Heidrich
KAUST
D. S. Jeon
KAIST
28
top related