Studying Dynamic Scenes with Time of Flight Cameras

Norbert Pfeifer, Wilfried Karel Sajid Ghuffar, Camillo Ressl norbert.pfeifer@geo.tuwien.ac.at

Research Group Photogrammetry

Department for Geodesy and Geoinformation Vienna University of Technology

geo.tuwien.ac.at

Images: Diploma thesis Stefan Niedermayer

Outline

Time of Flight Cameras Calibration and Scattering

still and dynamic imagery Optical Flow and Range Flow Range video analysis

dynamic imagery

General question: does high (temporal) sampling compensate for large random noise ?

2

ToF cameras: combining advantages

Area wise, simultaneous, robust determination of 3D object coordinates by range measurement of an emitted signal

• Suited for low contrast surfaces and in darkness • No homologous points necessary

No moving parts Low power consumption Small, compact, mobil Cheap Bundle of vectors

• Simplifies indirect orientation on moving platforms

• Simultaneous capturing of a dynamic object space

3

... of photography and laser scanning

Distance measurement

d = ½ c t Common for all methods:

• nIR • Amplitude modulation

Pulsed modulation • Avalange photo diodes (APD) for detection • APD in Geiger mode: single photon counting

(SPAD) • Multiple Double Short time Integration (MDSI)

Continuous wave modulation (Photo mix detectors, PMD)

4

Inte

nsity

Time

PH

φ

A

λmod

Amplitude and Phase (distance) measured

PMD: typical technical data

Precision: ≅ cm Accuracy: ≅ dm

5

Swissranger

Image matrix 144x176px² Modulation 5-30MHz

Field of view 39.6°x47.5° Uniqueness range 30-5m

Max. framerate 25 fps Dimensions 50x67x42.5mm³

Illumination 1W Mass 162g

Carrier WL 850nm Focal distance 8mm

PMD: random range errors

Empirically determined σ2obsDist vs. average amplitude

σ2obsDist ∼ 1/A2

6

PMD: systematic range errors

Local errors • Distance itself (not linear) • Position at sensor • Amplitude • Reflectivity / material (?) • Integration time • Internal / external temperature • Incidence angle (?) • Temporal drift

Not local errors • Internal: scattering • External: multipath

Relations unclear

7

Scattering cmp. lens flare

Multipath

Distance calibration approach

Self calibration: model selection and parameter estimation as integrated process on one data set

Amplitudes: lower relative noise in comparison to range and low local deformation

Lateral resolution very low (sensor matrix) expoit image space entirely circular, non-coded targets

Planar test field: no multipath, little scattering Bundle block adjustment IOR & EOR EOR mask test field area and use only areas outside targets distance residuals

Check residuals against posssible factors of influence: model selection, parameter estimation by LS adjustment

8

δρ

Calibration with range videos

Low lateral resolution vs. high temporal resolution : 25kPixel vs. 25fps

• Image sequences • Handheld camera guidance

permanente, random movements, dense sampling of parameter space

• Automatic target tracking • ~ 6000 frames • Only distances < 2.5m :

– A ∼ 1/d2 – Motion blur (espcially rotations)

9

Calirbation with still images

Averaging of thousands of frames with EOR=const to „still“ image suited for entire distance range

Automatic target detection and orientation Laborious search for correspondence between

object space and image space (no target code) 850 „stills“: with difference in integration

time, exterior orientation changes in amplitude, object distance, position in image space, ...

2 test fields with different reflectivity separate between object distance and amplitude

10

Model selection and parameter estimation

11

0 1 2 3 4 5 6 7 8-0.1

-0.08

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

Observed dist. [m]

Der

. min

us o

bs. d

ist.

[m]

Der. minus obs. dist. corr. 4 all but obs. dist. , corr. modeloffset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs

0 1 2 3 4 5 6 7 80

2

4

6

8

10

12x 10

4

coun

t

0 2000 4000 6000 8000 10000 12000 14000 16000 18000-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

Amplitude []

Der

. min

us o

bs. d

ist.

[m]

Der. minus obs. dist. corr. 4 all but amplitude , corr. modeloffset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs

0 2000 4000 6000 8000 10000 12000 14000 16000 180000

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5x 10

5

coun

t

10 30 60 90 120 150 180 210 255-0.005

0

0.005

0.01

0.015

0.02

0.025

0.03

0.035

int. time []

der.

min

us o

bs. d

ist.

[m]

Der. minus obs. dist. corr. 4 all but int. time , corr. modeloffset,d1,d2s,d2c,d3s,d3c,A1,iT1,Row1,Row2,Col1,Col2,RowCol2 / origObs

Scattering

Internal relections Observation: mixture of focussed and scattered light Emphasized with high image contrast – active illumination Modell: addition in the complex plane

• Assumption: sinusoidal signal May produce „strange“ distances (phase wraps) Investigation of scattering

• As few assumptions as possible • Without prior calibration • Influencing factors

– Integration time? – Amplitude? – Distance? – Position at sensor?

12

Method

Background subtraction • Frames with / without foreground object • Vary parameters to be investigated • Background

– Black cardboard • Foreground

– Circular white target – Mounted on vertical staff

• Mask area of staff and half shadow (illumination not an ideal point source), investigation of remaining image area

2 types of differences: • Separated difference of amplitude & range → errors in the observation • Difference of the complex signal → error in the signal

High noise level • Static scene and fixed EOR • Pixel wise averaging over up to 60 hours of continuous data acquisition

13

a_0016: mean amp. diff [%]; iTime: 30

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

Variation of scattering with the position on the sensor

Complex subtraction Not shift invariant Radially symmetric to principal point

14

Range error

Findings of the experiment

Scattering linear in the amplitude double target size ~ double amplitude difference

Scattering additive in complex signal, independent of object space distance ... Internal effect

Varies with position on sensor Point spread function (using an unresolved target)

• 63% focussed light; 8 neighboring pixels : 1,4‰; 72% within disk (r=19px); side lobes (maxima) : 0,07‰

15

PSF: Application

16

𝑔𝑔 = 𝑓𝑓 ∗ ℎ + 𝜂𝜂 Observed image is a convolution of the “real” image with the PSF + noise

Inversion: Richardson-Lucy:

• 𝑓𝑓 𝑛𝑛+1 = 𝑓𝑓 𝑛𝑛 𝑔𝑔�̂�𝑓 𝑛𝑛 ∗ℎ

∗ ℎ𝑇𝑇

Reduce RMSE distance • spatially variant PSF:

73% • spatially invariant PSF:

69%

Amplitude and Range difference images with/without object in foreground before and after deconvolution

Analyzing dynamic scene content

Considerations Video sequences: image of object points move in data stream

vs. wide baseline: points „jump“ Range and brightness are functions parameterized over image space

r(x,y), i(x,y) Assumption: functions are continuous and differentiable

Brightness of points remains constant Wrong assumption because of active illumination (1/r2, cos(incidenceA))

Therefore Track range and brightness changes of points in image space for Computing camera trajectory (exterior orientation) Follow moving objects throughout the scene

17

Optical flow and range flow

Comparable least squares matching / ICP but approx. values = 0, no iterations, no re-assignment of correspondence

Apply to entire image area Range and brightness should be considered together (complementary?)

Use framework of optical flow and range flow

18

Intensity Image Depth Image Intensity Derivatives Depth derivatives

Optical and range flow

Equation for one pixel Two/three unknowns (optical/range flow) Image/surface gradients required, otherwise coefficients are zero Application in window – local method,

parameter estimation by least squares adjustment Applied to all pixels, e.g. independently

19

x

I(x) t1

t2

𝜕𝜕𝐼𝐼𝜕𝜕𝑥𝑥

𝜕𝜕𝑥𝑥𝜕𝜕𝑡𝑡

= −𝜕𝜕𝐼𝐼𝜕𝜕𝑡𝑡

𝐼𝐼𝑥𝑥𝑢𝑢 + 𝐼𝐼𝑦𝑦𝑣𝑣 + 𝐼𝐼𝑡𝑡 = 0

u

X(x)

Z(x) t1 t2

(U,W)

𝑍𝑍𝑥𝑥𝑈𝑈 + 𝑍𝑍𝑦𝑦𝑉𝑉 −𝑊𝑊 + 𝑍𝑍𝑡𝑡 = 0

Fusion of Range and Intensity for object tracking

20

𝐼𝐼𝑥𝑥u + 𝐼𝐼𝑦𝑦v + 𝐼𝐼𝑡𝑡 = 0 𝑍𝑍𝑥𝑥U+ 𝑍𝑍𝑦𝑦V − W + 𝑍𝑍𝑡𝑡 = 0

Optical Flow Constraint Equation Range Flow Constraint Equation

Intensity Image Depth Image Intensity Derivatives Depth derivatives

𝐼𝐼𝑥𝑥𝑓𝑓𝑍𝑍

𝐼𝐼𝑦𝑦𝑓𝑓𝑍𝑍

−𝐼𝐼𝑥𝑥𝑥𝑥𝑍𝑍− 𝐼𝐼𝑦𝑦

𝑦𝑦𝑍𝑍

𝑍𝑍𝑥𝑥𝑓𝑓𝑍𝑍

𝑍𝑍𝑦𝑦𝑓𝑓𝑍𝑍

−𝑍𝑍𝑥𝑥𝑥𝑥𝑍𝑍− 𝑍𝑍𝑦𝑦

𝑦𝑦𝑍𝑍− 1

𝑈𝑈𝑉𝑉𝑊𝑊

= −𝐼𝐼𝑡𝑡−𝑍𝑍𝑡𝑡

Long Term Trajectories

21

Independently moving objects • Long range trajectory generation • Spatio-temporal segmentation into independently moving object

22

Note noise level in depth and intensity image range errors at dark trousers of second person: calibration not applied missing texture around feet of first person: neither in range nor in brightness hardly background texture

movement within floor/wall plane pair depends on small sign at the wall: spatial regularization to determine full flow at all pixel

Video

23

Note • Illumination fall off • Systematic range errors • Full 3D estimation of flow, esp. lower train • Successful suppresion of erroneous background motion • Upper train shows consistent flow despite

grey level differences and range errors

Optical and Range Flow for Camera Relative Orientation

Motion fields generated by camera motion Estimating 6 parameters of relative orientation

using dense intensity and range data

24

Video

25

Add robust parameter estimation (reweighting in LS adjustment) to eliminate moving objects

Weights in LS adjustment

26

1 →0

Conclusions and findings

Calibration is possible and shows stability Systematic range errors in cm-dm order Stills provided better results in calibration than videos

• averaging of stills reduced random error and thus eased model selection • motion blur as additional source of signal (error) in dynamic scenarios

Internal scattering can become larger (dm-order) Internal scattering compensation possible but too time consuming

hardware solution to reduce internal scattering of light (better absorption) Combined optical and range flow as natural combination of both channels

can consider stochastic signal properties, as it is embedded in LSA High noise level propagates to flow vectors due to local analysis

e.g. fluctuation of flow vectors, boundaries of segments Random noise can only be effectively removed with very rigid models ToF cameras rather trigger method than application development ( Quantitative evaluation, details on methods, etc.: in the publications )

27

41

42

Ende

43