Achuta Kadambi

MIT Media Lab

2

Macroscopic InterferometryAchuta Kadambi, MIT Media Lab

Joint work with Jamie Schiel and Ramesh Raskar

Achuta Kadambi

MIT Media Lab

3

Computer Vision/Graphics

Achuta Kadambi

MIT Media Lab

4

Computer Vision/Graphics Real-Life

Electric field only shown

700 nanometer wavelength

Achuta Kadambi

MIT Media Lab

5

Computer Vision/Graphics Real-Life

Electric field only shown

700 nanometer wavelength

Microscopic wave phenomena Macro scenes

Achuta Kadambi

MIT Media Lab

6

Multi-depth 3D Camera

Achuta Kadambi

MIT Media Lab

7

Multi-depth 3D Camera

10 meter scene

Achuta Kadambi

MIT Media Lab

8

Multi-depth 3D Camera

10 meter scene

Performance competitive low SNR

Achuta Kadambi

MIT Media Lab

9

Previous Work

ICCV15: Uses wave phenomena of polarization to enhance 3D shape

Kadambi et al. ICCV 2015

Achuta Kadambi

MIT Media Lab

10

Previous Work

ICCV15: Uses wave phenomena of polarization to enhance 3D shape

Today: Use wave phenomena to create multi-depth 3D

Kadambi et al. ICCV 2015

Achuta Kadambi

MIT Media Lab

11

Scene

A

BA

B

Achuta Kadambi

MIT Media Lab

12

Scene

A

BA

B

AI I

Achuta Kadambi

MIT Media Lab

13

Scene

A

BA

B

BI I

Achuta Kadambi

MIT Media Lab

14

Scene

A

BA

B

Achuta Kadambi

MIT Media Lab

15

Scene

A

BA

B

A BI I I

Achuta Kadambi

MIT Media Lab

16

Scene

A

BA

B

Achuta Kadambi

MIT Media Lab

17

Scene

A

BA

B

Interference Term

co2 sA B A BI II I I

Achuta Kadambi

MIT Media Lab

18

Scene

A

BA

B

Coherent Source

Interference graphic from sciencetalenter.dk

Interference Term

co2 sA B A BI II I I

Achuta Kadambi

MIT Media Lab

19

InterferometryIntentionally superimposing light to get scene clues

Figure from ligo.Caltech.edu

Achuta Kadambi

MIT Media Lab

20

Interferometry uses a reference to probe the scene

Reference Signal

Sample

Signal

Phase Delay

Achuta Kadambi

MIT Media Lab

21

Tiny Wavelength Good for Microscopic Scenes

Reference Signal

Sample

Signal

Phase Delay

700 nanometer wavelength

Achuta Kadambi

MIT Media Lab

22

Tiny Wavelength Good for Microscopic Scenes

Reference Signal

Sample

Signal

Phase Delay

700 nanometer wavelength

Corneal OCT

Corneal OCT from Carl Zeiss Visante ®

Achuta Kadambi

MIT Media Lab

23

Tiny Wavelength Good for Microscopic Scenes

Reference Signal

Sample

Signal

Phase Delay

700 nanometer wavelength

Corneal OCT

Corneal OCT from Carl Zeiss Visante ®

Achuta Kadambi

MIT Media Lab

24

Smaller the measurement ..

LIGO Project to Detect Grav Waves

Measure sample deviations less than the size of a proton

Achuta Kadambi

MIT Media Lab

25

Smaller the measurement .. Bigger the instrument

LIGO Project to Detect Grav Waves

Measure sample deviations less than the size of a proton

.. But kilometers in size

Achuta Kadambi

MIT Media Lab

26

Vision goal: small device, large scene

Achuta Kadambi

MIT Media Lab

27

Vision goal: small device, large scene

700 nanometer wavelength

Optical Phase

Achuta Kadambi

MIT Media Lab

28

Vision goal: small device, large scene

700 nanometer wavelength

Optical Phase Electronic Phase

Kinect uses electronic phase

Achuta Kadambi

MIT Media Lab

29

Terminology

Primal-Domain: Original Frame a Signal is Sampled In

Dual-Domain: Frequency transform with respect to Primal

Phase ToF: existing Kinect-style ToF cameras

Achuta Kadambi

MIT Media Lab

30

Phase ToF (Kinect)

Achuta Kadambi

MIT Media Lab

31

Phase ToF (Kinect)

2 M

zc

f

Achuta Kadambi

MIT Media Lab

32

How the Phase ToF (Kinect) Works

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1Signal One and Signal Two

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 100000

1

2

3

4

5

6

7

8

9

10x 10

7 Fourier Transform of Cross Correlation Function dt = 0.0001;

N = 20000;

t = 0:dt:(N-1)*dt;

phi1 = pi/3; phi2 = 0;

s1 = sin(2*pi*1*t + phi1);

s2 = sin(2*pi*1*t+ phi2);

xcFFT = conj( fft(s1) ) .*

fft(s2);

[~, fundamental_idx] =

max(xcFFT);

phase_difference = angle(

xcFFT(fundamental_idx) );

Achuta Kadambi

MIT Media Lab

33

Block Diagram of Phase ToF

cos M ts t f

cos Mr t f t

Primal-domain is \tau

For clarity, all constants removed

Phase ToF Refer.

Phase ToF Sample

Achuta Kadambi

MIT Media Lab

34

Block Diagram of Phase ToF

Phase ToF Refer.

cos M ts t f

cos Mr t f t

For clarity, all constants removed

cos Mc f

Phase ToF Xcorr

Phase ToF Sample

Achuta Kadambi

MIT Media Lab

35

Compute FFT wrt.

& Take angle of fundam.

Block Diagram of Phase ToF

Phase ToF Refer.

Phase ToF Sample

cos M ts t f

cos Mr t f t

Primal-domain is \tau

For clarity, all constants removed

cos Mc f

Phase ToF Xcorr

Achuta Kadambi

MIT Media Lab

36

Problems with Phase ToF

Reflections of two or more sine waves come back and mix

Multipath Interference

Phase Wrapping

Phase is periodic – long-range depths will wrap

Low SNR

Calculating phase requires multiple steps (prop. of errors)

Achuta Kadambi

MIT Media Lab

37

Computational ToF Imaging

By count of recent papers, appears to be a hot area in comp. photo

[Heide and Hullin et al. 2013]

[Godbaz et al. 2008]

[Kadambi et al. 2013]

[Bhandari et al. 2014]

[Jayasuriya et al. 2015]

[Gkioulekas et al. 2015]

Find more details at ICCV 2015 course:

www.media.mit.edu/~achoo/iccvtoftutorial/

[Su et al. 2016] -- @ Wed AM posters

[Shreshtha et al. 2016] -- @ next month’s SIGGRAPH

[Tsai et al. 2016]

Achuta Kadambi

MIT Media Lab

38

Computational ToF Imaging

By count of recent papers, appears to be a hot area in comp. photo

[Heide and Hullin et al. 2013]

[Godbaz et al. 2008]

[Kadambi et al. 2013]

[Bhandari et al. 2014]

[Jayasuriya et al. 2015]

[Gkioulekas et al. 2015]

Find more details at ICCV 2015 course:

www.media.mit.edu/~achoo/iccvtoftutorial/

[Su et al. 2016] -- @ Wed AM posters

[Shreshtha et al. 2016] -- @ next month’s SIGGRAPH

[Tsai et al. 2016]

Our Contribution: change primal-domain

Achuta Kadambi

MIT Media Lab

39

Changing the Primal Domain

Compute FFT wrt.

Primal-domain is \tau

For clarity, all constants removed

cos Mc f

Phase ToF Xcorr

Phase ToF Refer.

Phase ToF Sample

cos M ts t f

cos Mr t f t

Achuta Kadambi

MIT Media Lab

40

Changing the Primal Domain

cos Mc f

Phase ToF Xcorr

For clarity, all constants removed

Achuta Kadambi

MIT Media Lab

41

Changing the Primal Domain

cos Mc f

Phase ToF Xcorr

For clarity, all constants removed

Achuta Kadambi

MIT Media Lab

42

Changing the Primal Domain

cos Mc f

Phase ToF Xcorr

For clarity, all constants removed

2 M

zc

f

Achuta Kadambi

MIT Media Lab

43

Changing the Primal Domain

cos Mc f

Phase ToF Xcorr

For clarity, all constants removed

2 M

zc

f

cos2

M Mc ffc

z

Achuta Kadambi

MIT Media Lab

44

Changing the Primal Domain

cos Mc f

Phase ToF Xcorr

For clarity, all constants removed

2 M

zc

f

cos2

M Mc ffc

z

, cos2

M M M

zc f

cff

Achuta Kadambi

MIT Media Lab

45

Changing the Primal Domain

cos Mc f

Phase ToF Xcorr

For clarity, all constants removed

2 M

zc

f

cos2

M Mc ffc

z

, cos2

M M M

zc f

cff

2

00, cosM M M

zc ff f

c

Achuta Kadambi

MIT Media Lab

46

Changing the Primal Domain

cos Mc f

Phase ToF Xcorr

For clarity, all constants removed

2 M

zc

f

cos2

M Mc ffc

z

, cos2

M M M

zc f

cff

cos2

M M

zc f f

c

2

00, cosM M M

zc ff f

c

Achuta Kadambi

MIT Media Lab

47

Changing the Primal Domain

cos Mc f

Phase ToF Xcorr

For clarity, all constants removed

2 M

zc

f

cos2

M Mc ffc

z

, cos2

M M M

zc f

cff

cos2

M M

zc f f

c

2

00, cosM M M

zc ff f

c

Frequency ToF Xcorr

Achuta Kadambi

MIT Media Lab

48

Changing the Primal Domain

cos Mc f

Phase ToF Xcorr

For clarity, all constants removed

2 M

zc

f

cos2

M Mc ffc

z

, cos2

M M M

zc f

cff

cos2

M M

zc f f

c

2

00, cosM M M

zc ff f

c

Primal-domain is f_M

Dual-domain corresponds

to pathlength.

Frequency ToF Xcorr

Achuta Kadambi

MIT Media Lab

49

Achuta Kadambi

MIT Media Lab

50

Achuta Kadambi

MIT Media Lab

51

Macroscopic Scenes

Achuta Kadambi

MIT Media Lab

52

Macroscopic Scenes

Achuta Kadambi

MIT Media Lab

53

Macroscopic Scenes

Achuta Kadambi

MIT Media Lab

54

Multi-depth 3D Camera

Need only 4 frequencies to recover both reflections

Achuta Kadambi

MIT Media Lab

55

Multi-depth 3D Camera

Need only 4 frequencies to recover both reflections

Achuta Kadambi

MIT Media Lab

56

Multi-depth 3D Camera

Need only 4 frequencies to recover both reflections

Achuta Kadambi

MIT Media Lab

57

Resolving Multipath Interference

Scene

Achuta Kadambi

MIT Media Lab

58

How close can the reflections be?

Achuta Kadambi

MIT Media Lab

59

How close can the reflections be?

Achuta Kadambi

MIT Media Lab

60

Analyzing multipath estimation in low SNR

Achuta Kadambi

MIT Media Lab

61

Conclusion

Formalize link between ToF cameras and optical interferometry

Changed primal-domain for multi-depth, low SNR, long-range 3D

achoo@mit.eduwww.media.mit.edu/~achoo

Collaborators

Jamie SchielRamesh Raskar

Acknowledgements:

Ayush BhandariSuren JayasuriyaVage Taamazyan