extended depth of field and broadband imaging with ...clim.inria.fr/workshop/sahin-clim.pdf ·...
Post on 16-Aug-2020
3 Views
Preview:
TRANSCRIPT
Extended depth of field and broadband imaging with
diffractive optical elements
Erdem Sahin*, Ugur Akpinar, Atanas Gotchev
Tampere University, Finland
*erdem.sahin@tuni.fi
Outline
• Problem 1: Extended depth of field (EDoF) imaging
• Phase-coded computational camera: refractive lens and DOE
• Problem 2: EDoF and broadband imaging with single DOE
• DOE-only computational camera
Problem 1: Extended-DoF imaging
?
Blurred image
(Shallow-DoF)
Computational
deblurring
Sharp image
(Extended-DoF)
𝐼𝑠 𝐼𝑜𝐼𝑜 = 𝑓(𝐼𝑠)
Extended-DoF
DoF
Phase maskThin lens Sensor
?
DoF
Thin lens Sensor
CameraComputational
Camera
Method: Joint optimization of end-to-end system
Method: Computational camera modelPhase mask
Extended-DoF
𝑠 𝑥𝑧𝑖𝑧𝑒+ 𝑧𝑒
−
PSFℎ𝜆,𝑧(𝑥, 𝑦)
Φ𝜆(𝑠, 𝑡)
𝑧 = 𝑧𝑓
𝑟
Generalized pupil function (GPF)
Point spread function (PSF) Sensor image
Defocus coefficient
• Chromatic aberration (dispersion): Refractive vs diffractive lens, assume 𝜆0 nominal wavelength:
• Refractive (plano-convex) lens: exp−𝑗 𝜋
𝜆𝑓𝜆𝑠2 + 𝑡2 ; 𝑓𝜆= 𝑓𝜆0
𝑛𝜆0−1
𝒏𝝀−1(Material dispersion)
• DOE lens: exp−𝑗2 𝜋
𝜆(𝑛𝜆−1)𝑑(𝑠, 𝑡) ; 𝑑 𝑠, 𝑡 = 𝑚𝑜𝑑
(𝑠2+𝑡2)
2𝑓𝜆0, 𝜆0 ; 𝑓𝜆
𝐷𝑂𝐸=𝑓𝜆0𝜆0
𝜆(Diff. dispersion; multiple 𝑓’s)
• We train the network using RGB images assigning each channel to a single wavelength, 𝜆 ∈ {𝑅, 𝐺, 𝐵}, and optimize Φ𝜆0 𝑠, 𝑡 , i.e., equivalently, 𝑑 𝑠, 𝑡 .
Method: Computational camera model
Refractive lens DOE lens
Φ𝜆 𝑠, 𝑡 = Φ𝜆0 𝑠, 𝑡𝜆0 𝑛𝜆 − 1
𝜆(𝑛𝜆0 − 1)
• Mask sampling rate (signal space of the phase mask)?
• ~ 3𝜇m (lithographic fabrication): do we need this high resolution?
Method: Computational camera model
D: defocus term phase (mask) modulation DoF range:
GPF~MTF
Method: Deblurring-CNN
Conv
d=1
BN
ReLU
𝐼𝑆Conv
d=2
BN
ReLU
Conv
d=3
BN
ReLU
Conv
d=2
BN
ReLU
Conv
d=1
BN
ReLU
ConvConvT
ReLUConvT+ 𝐼𝑜
𝑁𝑥 × 𝑁𝑦 × 3 𝑁𝑥 × 𝑁𝑦 × 32 𝑁𝑥 × 𝑁𝑦 × 3
Loss:
Results
𝑠 𝑥
𝑇 = 5𝑚𝑚𝑓𝜆0 = 34.9𝑚𝑚 𝑓/7
𝜆0 = 625𝑛𝑚
Δ𝑠𝑜𝑝𝑡
≈ 20𝜇𝑚; Δ𝑓𝑎𝑏 = 20𝜇𝑚𝜆 = 625𝑛𝑚, 535𝑛𝑚, 445𝑛𝑚
𝑛 ∼ 𝑁(0, 𝜎2)𝜎 ∼ 𝑈 0.001, 0,01
Δ𝑥 = 2𝜇𝑚
𝑧𝑓 = 2𝑚
Defocus: Ψ =𝜋𝑇2
4𝜆
1
𝑧+
1
𝑧𝑓, 𝚿 ∈ −𝟒𝟎, 𝟓𝟎
Extended-DoF
𝑧𝑒− = 0.5𝑚(2D)
𝑧𝑒+ = ∞(0D)
BSDS500 dataset[1]
400 training (300 × 300 patches), 100 test images
𝑧 = 𝑈[0D, 2D]
[1] https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/asd
Results Optimized height map
Results: RGB images
[1] Edward R. Dowski and W. Thomas Cathey. 1995. Extended depth of field through wave-front coding. OSA Appl. Opt. 34, 11 (1995), 1859–1866.
[2] V. Sitzmann, S. Diamond, Y. Peng, X. Dun, S. Boyd, W. Heidrich, F. Heide, and G. Wetzstein. 2018. End-to-end optimization of optics and image processing for
achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37, 4, 2018
*U. Akpinar, E. Sahin, A. Gotchev. Learning Optimal Phase-coded Aperture for Depth of Field Extension. IEEE International Conference on Image Processing, 2019.
(sensor image)*
Results: RGB images, noise, 1m (𝑧𝑓𝐺)
𝜎𝑡𝑟𝑎𝑖𝑛 ∼ 𝑈 0.001, 0,01
Sensor
Image
Network
Output
Results: lens only vs DOE+lens, 0.5m (𝑧𝑒−)
Refractive lens only DOE+lens
14.79 24.59PSNR (dB)
Ground Truth
Sensor
Image
Network
Output
• The same D-CNN is optimized for
both cases seperately.
Results: Multispectral input (real life)
Assumed sensor response[2]
Multispectral data[1]
500 × 500 × 31 resolution
𝜆 ∈ 420 − 720 𝑛𝑚
𝜆
𝑥𝑦
[1] Monno, Yusukex, et al. "A practical one-shot multispectral imaging system using a single image sensor." IEEE Transactions on Image Processing 24.10 (2015): 3048-3059.
[2] Kodak, “KAF-10500 Image Sensor,” KAF-10500-CXA-JH-AE datasheet, [Revised July 2007], Available: https://www.datasheets360.com/pdf/4613689109339751409
Refractive lens only DOE+lens
Sensor
image
Network
output
PSNR (dB) 37.67 37.62
Ground Truth
Results: Multispectral input, 1m (𝑧𝑓𝐺)
Refractive lens only DOE+lens
Sensor
image
Network
output
PSNR (dB) 26.70 34.02
Ground Truth
Results: Multispectral input, 0.5m (𝑧𝑒−)
Problem 2: EDoF and broadband imaging with single DOE
?
Blurred image
with chromatic
abberation
Computational
deblurring &
chromatic
aberration
correction
Sharp image
without
chromatic
aberration
𝐼𝑠 𝐼𝑜𝐼𝑜 = 𝑓(𝐼𝑠)
Extended-DoF
DoF
DOESensor
?
Computational
camera
Method (same): Joint optimization of end-to-end system
Method: Computational camera model, DOE-only
Phase mask
Extended-DoF
𝑠 𝑥𝑧𝑖𝑧𝑒+ 𝑧𝑒
−
PSFℎ𝜆,𝑧(𝑥, 𝑦)
Φ𝜆(𝑠, 𝑡)
𝑧 = 𝑧𝑓
𝑟
• Chromatic aberration (dispersion):
• Hybrid refractive lens and DOE system: automatically correct some of the aberration itself.
• DOE-only case: chromatic aberration correction is more challenging!
Method: Computational camera model, DOE-only
DOE lens+phaseRefractive lens DOE phase
• Mask sampling rate (signal space of the phase mask)?
• ~ 3𝜇m (lithographic fabrication): do we need this high resolution?
Method: Computational camera model, DOE-only
• High BW: Δ𝑠 ≤2𝜆𝑓 𝜆
𝑟
Δ𝑠 ≤ 3𝜇𝑚,
for 𝑇 = 5𝑚𝑚, 𝜆 = 534𝑛𝑚, 𝑓𝜆 = 35𝑚𝑚
• Need even higher BWs for larger NAs!Refractive lens:
DOE-only:
Method (same): Deblurring-CNN, DOE-only
Conv
d=1
BN
ReLU
𝐼𝑆Conv
d=2
BN
ReLU
Conv
d=3
BN
ReLU
Conv
d=2
BN
ReLU
Conv
d=1
BN
ReLU
ConvConvT
ReLUConvT+ 𝐼𝑜
𝑁𝑥 × 𝑁𝑦 × 3 𝑁𝑥 × 𝑁𝑦 × 32 𝑁𝑥 × 𝑁𝑦 × 3
Loss:
• Larger size (resolution) PSF (>1000): convergence?
Results: RGB images, DOE-only
z=0.5m
z=1m
PSNR (dB)
29.41
23.77
Sensor image Network output Ground truth (object)
Results: Multispectral input, DOE-only
Sensor
image
Network
output
PSNR (dB) 23.99 23.85
Ground Truth
z=1m z=0.5m
Results: Refractive lens+DOE vs DOE-only
Refractive lens+DOE DOE-only
Conclusions
• Phase-coded computational camera: jointly optimized DOE and D-CNN can provide significant DoF extension.
• Definition of signal space (DOE resolution) is critical for efficient convergence of end-to-end network.
• EDoF and broadband imaging with DOE-only optics is challenging:
• The (D-CNN) network should be further improved to tackle with chromatic aberrations
• The signal space of the DOE (hence PSF) should be cleverly defined, e.g., based on EDoF
• Training with multispectral images
• Similar framework can be applied to other problems: task specific computational cameras.
top related