Extended depth of field and broadband imaging with

diffractive optical elements

Erdem Sahin*, Ugur Akpinar, Atanas Gotchev

Tampere University, Finland

*erdem.sahin@tuni.fi

Outline

• Problem 1: Extended depth of field (EDoF) imaging

• Phase-coded computational camera: refractive lens and DOE

• Problem 2: EDoF and broadband imaging with single DOE

• DOE-only computational camera

Problem 1: Extended-DoF imaging

?

Blurred image

(Shallow-DoF)

Computational

deblurring

Sharp image

(Extended-DoF)

𝐼𝑠 𝐼𝑜𝐼𝑜 = 𝑓(𝐼𝑠)

Extended-DoF

DoF

Phase maskThin lens Sensor

?

DoF

Thin lens Sensor

CameraComputational

Camera

Method: Joint optimization of end-to-end system

Method: Computational camera modelPhase mask

Extended-DoF

𝑠 𝑥𝑧𝑖𝑧𝑒+ 𝑧𝑒

−

PSFℎ𝜆,𝑧(𝑥, 𝑦)

Φ𝜆(𝑠, 𝑡)

𝑧 = 𝑧𝑓

𝑟

Generalized pupil function (GPF)

Point spread function (PSF) Sensor image

Defocus coefficient

• Chromatic aberration (dispersion): Refractive vs diffractive lens, assume 𝜆0 nominal wavelength:

• Refractive (plano-convex) lens: exp−𝑗 𝜋

𝜆𝑓𝜆𝑠2 + 𝑡2 ; 𝑓𝜆= 𝑓𝜆0

𝑛𝜆0−1

𝒏𝝀−1(Material dispersion)

• DOE lens: exp−𝑗2 𝜋

𝜆(𝑛𝜆−1)𝑑(𝑠, 𝑡) ; 𝑑 𝑠, 𝑡 = 𝑚𝑜𝑑

(𝑠2+𝑡2)

2𝑓𝜆0, 𝜆0 ; 𝑓𝜆

𝐷𝑂𝐸=𝑓𝜆0𝜆0

𝜆(Diff. dispersion; multiple 𝑓’s)

• We train the network using RGB images assigning each channel to a single wavelength, 𝜆 ∈ {𝑅, 𝐺, 𝐵}, and optimize Φ𝜆0 𝑠, 𝑡 , i.e., equivalently, 𝑑 𝑠, 𝑡 .

Method: Computational camera model

Refractive lens DOE lens

Φ𝜆 𝑠, 𝑡 = Φ𝜆0 𝑠, 𝑡𝜆0 𝑛𝜆 − 1

𝜆(𝑛𝜆0 − 1)

• Mask sampling rate (signal space of the phase mask)?

• ~ 3𝜇m (lithographic fabrication): do we need this high resolution?

Method: Computational camera model

D: defocus term phase (mask) modulation DoF range:

GPF~MTF

Method: Deblurring-CNN

Conv

d=1

BN

ReLU

𝐼𝑆Conv

d=2

BN

ReLU

Conv

d=3

BN

ReLU

Conv

d=2

BN

ReLU

Conv

d=1

BN

ReLU

ConvConvT

ReLUConvT+ 𝐼𝑜

𝑁𝑥 × 𝑁𝑦 × 3 𝑁𝑥 × 𝑁𝑦 × 32 𝑁𝑥 × 𝑁𝑦 × 3

Loss:

Results

𝑠 𝑥

𝑇 = 5𝑚𝑚𝑓𝜆0 = 34.9𝑚𝑚 𝑓/7

𝜆0 = 625𝑛𝑚

Δ𝑠𝑜𝑝𝑡

≈ 20𝜇𝑚; Δ𝑓𝑎𝑏 = 20𝜇𝑚𝜆 = 625𝑛𝑚, 535𝑛𝑚, 445𝑛𝑚

𝑛 ∼ 𝑁(0, 𝜎2)𝜎 ∼ 𝑈 0.001, 0,01

Δ𝑥 = 2𝜇𝑚

𝑧𝑓 = 2𝑚

Defocus: Ψ =𝜋𝑇2

4𝜆

1

𝑧+

1

𝑧𝑓, 𝚿 ∈ −𝟒𝟎, 𝟓𝟎

Extended-DoF

𝑧𝑒− = 0.5𝑚(2D)

𝑧𝑒+ = ∞(0D)

BSDS500 dataset[1]

400 training (300 × 300 patches), 100 test images

𝑧 = 𝑈[0D, 2D]

[1] https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/asd

Results Optimized height map

Results: RGB images

[1] Edward R. Dowski and W. Thomas Cathey. 1995. Extended depth of field through wave-front coding. OSA Appl. Opt. 34, 11 (1995), 1859–1866.

[2] V. Sitzmann, S. Diamond, Y. Peng, X. Dun, S. Boyd, W. Heidrich, F. Heide, and G. Wetzstein. 2018. End-to-end optimization of optics and image processing for

achromatic extended depth of field and super-resolution imaging. ACM Trans. Graph. 37, 4, 2018

*U. Akpinar, E. Sahin, A. Gotchev. Learning Optimal Phase-coded Aperture for Depth of Field Extension. IEEE International Conference on Image Processing, 2019.

(sensor image)*

Results: RGB images, noise, 1m (𝑧𝑓𝐺)

𝜎𝑡𝑟𝑎𝑖𝑛 ∼ 𝑈 0.001, 0,01

Sensor

Image

Network

Output

Results: lens only vs DOE+lens, 0.5m (𝑧𝑒−)

Refractive lens only DOE+lens

14.79 24.59PSNR (dB)

Ground Truth

Sensor

Image

Network

Output

• The same D-CNN is optimized for

both cases seperately.

Results: Multispectral input (real life)

Assumed sensor response[2]

Multispectral data[1]

500 × 500 × 31 resolution

𝜆 ∈ 420 − 720 𝑛𝑚

𝜆

𝑥𝑦

[1] Monno, Yusukex, et al. "A practical one-shot multispectral imaging system using a single image sensor." IEEE Transactions on Image Processing 24.10 (2015): 3048-3059.

[2] Kodak, “KAF-10500 Image Sensor,” KAF-10500-CXA-JH-AE datasheet, [Revised July 2007], Available: https://www.datasheets360.com/pdf/4613689109339751409

Refractive lens only DOE+lens

Sensor

image

Network

output

PSNR (dB) 37.67 37.62

Ground Truth

Results: Multispectral input, 1m (𝑧𝑓𝐺)

Refractive lens only DOE+lens

Sensor

image

Network

output

PSNR (dB) 26.70 34.02

Ground Truth

Results: Multispectral input, 0.5m (𝑧𝑒−)

Problem 2: EDoF and broadband imaging with single DOE

?

Blurred image

with chromatic

abberation

Computational

deblurring &

chromatic

aberration

correction

Sharp image

without

chromatic

aberration

𝐼𝑠 𝐼𝑜𝐼𝑜 = 𝑓(𝐼𝑠)

Extended-DoF

DoF

DOESensor

?

Computational

camera

Method (same): Joint optimization of end-to-end system

Method: Computational camera model, DOE-only

Phase mask

Extended-DoF

𝑠 𝑥𝑧𝑖𝑧𝑒+ 𝑧𝑒

−

PSFℎ𝜆,𝑧(𝑥, 𝑦)

Φ𝜆(𝑠, 𝑡)

𝑧 = 𝑧𝑓

𝑟

• Chromatic aberration (dispersion):

• Hybrid refractive lens and DOE system: automatically correct some of the aberration itself.

• DOE-only case: chromatic aberration correction is more challenging!

Method: Computational camera model, DOE-only

DOE lens+phaseRefractive lens DOE phase

• Mask sampling rate (signal space of the phase mask)?

• ~ 3𝜇m (lithographic fabrication): do we need this high resolution?

Method: Computational camera model, DOE-only

• High BW: Δ𝑠 ≤2𝜆𝑓 𝜆

𝑟

Δ𝑠 ≤ 3𝜇𝑚,

for 𝑇 = 5𝑚𝑚, 𝜆 = 534𝑛𝑚, 𝑓𝜆 = 35𝑚𝑚

• Need even higher BWs for larger NAs!Refractive lens:

DOE-only:

Method (same): Deblurring-CNN, DOE-only

Conv

d=1

BN

ReLU

𝐼𝑆Conv

d=2

BN

ReLU

Conv

d=3

BN

ReLU

Conv

d=2

BN

ReLU

Conv

d=1

BN

ReLU

ConvConvT

ReLUConvT+ 𝐼𝑜

𝑁𝑥 × 𝑁𝑦 × 3 𝑁𝑥 × 𝑁𝑦 × 32 𝑁𝑥 × 𝑁𝑦 × 3

Loss:

• Larger size (resolution) PSF (>1000): convergence?

Results: RGB images, DOE-only

z=0.5m

z=1m

PSNR (dB)

29.41

23.77

Sensor image Network output Ground truth (object)

Results: Multispectral input, DOE-only

Sensor

image

Network

output

PSNR (dB) 23.99 23.85

Ground Truth

z=1m z=0.5m

Results: Refractive lens+DOE vs DOE-only

Refractive lens+DOE DOE-only

Conclusions

• Phase-coded computational camera: jointly optimized DOE and D-CNN can provide significant DoF extension.

• Definition of signal space (DOE resolution) is critical for efficient convergence of end-to-end network.

• EDoF and broadband imaging with DOE-only optics is challenging:

• The (D-CNN) network should be further improved to tackle with chromatic aberrations

• The signal space of the DOE (hence PSF) should be cleverly defined, e.g., based on EDoF

• Training with multispectral images

• Similar framework can be applied to other problems: task specific computational cameras.