continuous 360 degree real-time 3d laser...
TRANSCRIPT
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 1
Continuous 360 degree real-time 3D laser scanner
Marko Reimer, Oliver Wulf, Bernardo Wagner
Research Centre L3S andInstitute for Systems Engineering
University of Hannover
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 2
Motivation
Localization for mobile robots
Path planning for mobile robots
Environment modeling for mobile robots
Goals: Unrestricted autonomous navigation and 3D map building in real-time
3D visualization for human operators
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 3
Why use 3D data for navigation?
Real indoor environments are often cluttered
Good landmarks like walls are often partially occluded and thus invisible for 2D sensors
Obstacles not within the 2D scanline may exist
Negative obstacles can not be detected in 2D scans
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 4
Ways to reach the 3.rd dimension
4 Alternative scanning methods
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 5
Main components of the used hardware
ScanDrive (rotating platform)Capable of continuous rotationNo accelerations
- Less mechanical load- Less power consumption
Data- and Power-Transfer via slip rings
Scalable Processing Box (SPB)Embedded PC running “LiRE” (Linux Realtime Environment) as real-time operating system
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 6
Result of real-time synchronisation
3D-laser scans consist of data fusion from multiple sensors
(2D laser scanner data, laser scanner orientation and robot position)
Systematic errors can be avoided by exact synchronization
Poor adjustment Good adjustment
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 7
Software / Sensor architecture
3D Scanner
2D Scanner Rotating plate Odometrie
Chassis3D Gyro
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 8
Real-time data~32.000 points in ~2.4 sec
Two main aspects encourage the use of a real-time operating system
All gathered data is marked with a synchronized timestamp
Later and offline processing is possible while keeping the data synchronized
Data can be gathered during movement
The data processing is always done within a specified time
The system acts like a sensor ->no higher-level system computations needed
Resulting data can be used for time critical operations
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 9
Visualization of the 3d data I
As depth image (gray values representing distance)
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 10
Visualization of the 3d data II
As 3D point cloud (already with colored objects)
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 11
3D Perception and 2D Maps
3D map representations
Registered 3D point clouds
Digital elevation maps (2½D)
Computational expensive
Combination of 3D perception and 2D map representation by use of Virtual 2D Scans
Full 3D information is not needed for robot navigation
Existing 2D navigation algorithms work more efficient
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 12
Complexity reduction
Map building requires landmarks (2D)
Path planning requires obstacle information. (2D)
=> Two “different maps” needed
Red points are detected landmarks
(virtual landmark scan)
Blue points are detected obstacles
(virtual obstacle scan)
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 13
Virtual 2D Scan Real Environment
Virtual 2D Scan3D-point cloud
09 Sep. 2005Marko Reimer 1. Range Imaging Research Day 2005 14
Designed a 3D laserscanner
For use on a mobile platform
Delivering 3D data in real-time
Selectable point density
Developed an
Fast
Efficient
Robust
algorithm for navigation of a mobile system.
Conclusion
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Robust 3D Measurement with PMD Sensors
“CCD goes 3D !”1st range imaging research day
8 September 2005, Zurich
CMOS
PMDTechnologies GmbHAm Eichenhang 50D-57076 Siegen
phone + 49 271 238538 -800fax + 49 271 238538 [email protected]
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
TheThe CompanyCompany
THE COMPANYTHE COMPANY
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Company Profile: PMD Technologies GmbHCompany Profile: PMD Technologies GmbH
- Founded in 2002
- Employees: 20+
- Shareholders: - Audi AG (50%) - ifm electronics (50%)
- Worldwide patents
- Markets: Automotive, Industrial and consumer.
- 1st all-solid-state-3D series-product worldwide
- Camera and chip prototypes available for evaluation of 3D technology
- Automotive: camera systems in field test with several OEMs.
- Awards:- „German future award“ 2002 nominee- Hermes award 2005 for “efector PMD”
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
BASIC PRINCIPLEBASIC PRINCIPLE
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
What makes 3DWhat makes 3D--PMD so smart?PMD so smart?
CCD or CMOS 2D-Imagers“Each pixel is a photodiode”
PMD“Each pixel is a complete distance
measurement system”
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
TTimeime--OOff--FFlight Basic Principlelight Basic Principle
SENDER
RECEIVER
00:00:00
Start
Stop
Distance is measured by determining the turn- around time a light pulse needs to travel from the sender to the target and back to the receiver. With knowledge of speed of light the distance can easily be calculated.
Distance is measured by determining the turnDistance is measured by determining the turn-- around time a light pulse needs to around time a light pulse needs to travel from the sender to the target and back to the receiver. Wtravel from the sender to the target and back to the receiver. With knowledge of ith knowledge of speed of light the distance can easily be calculated.speed of light the distance can easily be calculated.
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
3D TOF3D TOF-- CameraCamera
Chip measures both intensity and distance (arrival time of light) in each pixel 3D-camera
Chip measures both intensity and distance Chip measures both intensity and distance (arrival time of light) in each pixel (arrival time of light) in each pixel 3D3D--cameracamera
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
PMDPMD-- Principle: Basic Cross SectionPrinciple: Basic Cross Section
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
PMDPMD--operationoperation
d=0mPopt_sent
Popt_received
mod_A mod_B
Q_a Q_b
mod_A mod_B
Q_a Q_b
Mod_A, Mod_B A B
Q_A, Q_BA
B
d=2mPopt_sent
Popt_received
mod_A mod_B
Q_a Q_b
mod_A mod_B
Q_a Q_b
Mod_A, Mod_B A B
Q_A, Q_BA
B
d=3.75mPopt_sent
Popt_received
mod_A mod_B
Q_a Q_b
mod_A mod_B
Q_a Q_b
Mod_A, Mod_B A B
Q_A, Q_BA
B
d=5mPopt_sent
Popt_received
mod_A mod_B
Q_a Q_b
mod_A mod_B
Q_a Q_b
Mod_A, Mod_B A B
Q_A, Q_B
AB
d=7.5mPopt_sent
Popt_received
mod_A mod_B
Q_a Q_b
mod_A mod_B
Q_a Q_b
Mod_A, Mod_B A B
Q_A, Q_B
A
B
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Analysis of AKF (autocorrelation function)Analysis of AKF (autocorrelation function)
λ = 15 m fmod = 20MHz
τ ~ ϕ
Α1 Α2 Α3 Α4Α4
b
a
ϕ
AKF
Signal Phase ϕ: distanceSignal Phase Signal Phase ϕ: ϕ: distancedistance
Amplitude a: signal strength of active illumination: quality of 3D measurement.
Amplitude a: Amplitude a: signal strength of active illumination: signal strength of active illumination: quality quality of 3D measurement.of 3D measurement.
Offset b: gray scale imageOffset b: Offset b: gray scale imagegray scale image
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Key Innovation of PMDKey Innovation of PMD
1. Non-scanning 3D Arrays.Highly integrated Arrays of individual Range Finding Pixels possible. Low-cost, Robust
2. Classical TOF Range Finders drastically simplified.Complete Range Finder realized in quasi-optical Domain.No HF- Signal Processing.
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
CHALLENGESCHALLENGES
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
CHALLENGESCHALLENGES
Need to measure picoseconds. (What is a picosecond?)How much light do I need for a certain resolution?What impact has background light (sun)?
How bright is the sun?What are the problems?
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
What is a What is a picosecondpicosecond??
speed of light is 3E8 m/s = 2mm/6.67psTOF: 6.67ps resolution required for 1mm precisionspeed of light is 3E8 m/s = 2mm/6.67psTOF: 6.67ps resolution required for 1mm precision
1s 1E-3s 1E-6s 1E-9s 1E-12s
1s 1/1000s 300ps 6.67ps(3GHz) 1mm
1000 1000 1000 1000
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Predictable measurement precisionPredictable measurement precisionk: contrastS: # signal electronsk*S: # active (demodulated) electronsN: # noise electronsλmod: “length of modulation wave” (=c/fmod)
π⋅λ
⋅⋅
⋅=8N/Sk
1N
1dR mod
phase
What do we need for a high distance accuracy (low dR) ?
k: high high contrast PMD-pixels (+bandwidth), high contrast light source
S: high strong light source, large lens aperture, high pixel sensitivity (good fill factor, large pixels)good reflecting targets, close targets.
N: low low shot noise, low background light, low dark current low camera noise (system noise)
λmod: low (fmod high) high bandwidth.
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Predictable measurement precisionPredictable measurement precision
π⋅λ
⋅⋅
⋅=8N/Sk
1N
1dR mod
phase
0
0.5
1
1.5
2
2.5
3
3.5
4
50 100 150 200 250 300 350 400distance (cm)
stan
dard
dev
iatio
n (c
m)
PMD[vision] 19k MeasurementPMD[vision] 19k Theory
0
0.5
1
1.5
2
2.5
3
3.5
4
50 100 150 200 250 300 350 400distance (cm)
stan
dard
dev
iatio
n (c
m)
PMD[vision] 19k MeasurementPMD[vision] 19k Theory
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Influence of sunlightInfluence of sunlight
LED illumination
For most outdoor applications sunlight is much brighter than theactive illumination. This has two bad impacts on the TOF-system:
(1) Shot noise is increased ( distance noise)
(2) The dynamics of the sensor can not handle the sunlight (saturation).
For most outdoor applications sunlight is much brighter than theactive illumination. This has two bad impacts on the TOF-system:
(1) Shot noise is increased ( distance noise)
(2) The dynamics of the sensor can not handle the sunlight (saturation).
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Influence of sunlight Influence of sunlight –– simple workaroundssimple workarounds
LED illumination
Filter
Filter
LED illumination
LED illumination
Sunlight much brighter than LED illumination. Very poor SNR (active light / background light).
Sunlight much brighter than LED illumination. Very poor SNR (active light / background light).
Optical Spectral Filter:
Transmission only of LED- spectrum. Improvement of SNR by reduction (filtering) of background light.
Optical Spectral Filter:
Transmission only of LED- spectrum. Improvement of SNR by reduction (filtering) of background light.
Burst operation of LEDs:
Improvement of SNR by increase of active peak power.
Burst operation of LEDs:
Improvement of SNR by increase of active peak power.
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Calculation: Sunlight vs. Active lightCalculation: Sunlight vs. Active light
LED SUNLED 0.3 nm/KTemp. Automotive min -40 °CTemp. Automotive max 120 °CTemp. Automotive range 160 K
48 nm
LED 50 nmFilter 98 nm
Viewing angle 20 deg (diagonal)Range 40 mImage Format 1:1edge of image (object space) 10.0 marea of image (object space) 99 m^2#LEDS 100 LEDsoptical power per LED 0.04 W/LEDtotal optical power of LEDs (CW) 4 Wpower density in object space (CW) 0.040 W/m^2 726 W/m^2 power density of sunpower density in object space (CW, filtered) 0.039 W/m^2 83 W/m^2 power density of sun (filtered)Burst-Operation 5 Factor
20 Wpower desnisty in object space (Burst) 0.20 W/m^2power desnisty in object space (Burst, filtered) 0.19 W/m^2
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Situation 1: No Filter, No BurstSituation 1: No Filter, No Burst
0.01
0.1
1
10
100
1000
10000
400 500 600 700 800 900 1000 1100
wavelength in nm
optic
al p
ower
den
sity
in m
W/m
2/nm
sunLEDs (4W, 20deg FOV, 40m)
0.04W/m2
750W/m2
ratio sun/active = 18‘000 = 85dBratio sun/active = 18‘000 = 85dB
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Situation2: Spectral FilteringSituation2: Spectral Filtering
0.01
0.1
1
10
100
1000
10000
400 500 600 700 800 900 1000 1100wavelength in nm
optic
al p
ower
den
sity
in m
W/m
2/nm
0
10
20
30
40
50
60
70
80
90
100
%
sunLEDs (4W, 20deg FOV, 40m)sun, filterednormalized spectral response of PMDspectral filter
0.04W/m2
80W/m2 (improvement of factor 9)
ratio sun/active = 2‘000 = 66dBratio sun/active = 2‘000 = 66dB
750W/m2
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Situation 3: Spectral Filtering and Burst Operation of Situation 3: Spectral Filtering and Burst Operation of LEDsLEDs
0.01
0.1
1
10
100
1000
10000
400 500 600 700 800 900 1000 1100
wavelength in nm
optic
al p
ower
den
sity
in m
W/m
2/nm
0
10
20
30
40
50
60
70
80
90
100
%
sunLEDs (4W, 20deg FOV, 40m)sun, filteredLEDs (bursted, filtered)spectral filter
0.2W/m2 (factor 5)
83W/m2 (improvement of factor 9)
ratio sun/active = 400 = 52dBratio sun/active = 400 = 52dB
750W/m2
0.04W/m2
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Influence of sunlight Influence of sunlight -- SummarySummary
LED illumination
Filter
Filter
LED illumination
LED illumination
ratio of sunlight / active
18‘000 = 85dB
2‘000 = 66dB
Filter:-19dB
-33dB
Burst Operation-14dB
400 = 52dB
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Influence of sunlight Influence of sunlight -- SummarySummary
• Spectral Filtering and burst operation of light source are effective methods to improve „optical“ SNR.
• Reasonable sunlight rejection of these methods: -33dB.
• The remaining ratio of sunlight / active signal is still poor:
• Even the filtered sun is still a factor of 400 brighter than thepulsed LEDs.
Additional sunlight rejection methods have to be found.
• Spectral Filtering and burst operation of light source are effective methods to improve „optical“ SNR.
• Reasonable sunlight rejection of these methods: -33dB.
• The remaining ratio of sunlight / active signal is still poor:
• Even the filtered sun is still a factor of 400 brighter than thepulsed LEDs.
Additional sunlight rejection methods have to be found.
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Sum up: ChallengesSum up: Challenges
BandwidthSensitivityDynamics and sunlight robustness
… are the key challenges for optical TOF measurement.
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
TECHNICAL SOLUTIONSTECHNICAL SOLUTIONS
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Contrast Contrast –– measurementmeasurement
Bandwidth: beyond 100MHzBandwidth: beyond 100MHz
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Size of PMD Pixels is scalable.Size of PMD Pixels is scalable.
4040µµmm 118118µµmm 180180µµmm 280280µµmm
Due to the scalable size of PMD pixels, good fill factors (>50%) can be achieved, even when integrating additional pixel- functionality like SBI, A2I, Multi- sampling, …
Due to the scalable size of PMD pixels, good fill factors Due to the scalable size of PMD pixels, good fill factors (>50%) can be achieved, even when integrating additional (>50%) can be achieved, even when integrating additional pixelpixel-- functionality like SBI, A2I, Multifunctionality like SBI, A2I, Multi-- sampling, sampling, ……
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
SBI: Suppression of background illuminationSBI: Suppression of background illumination
SBI
Working Principle of SBI:
The SBI circuitry removes the charge packets of background illumination and dark current from the readout nodes. The complete dynamic range of the readout chain can be used for the active signal.
Working Principle of SBI:
The SBI circuitry removes the charge packets of background illumination and dark current from the readout nodes. The complete dynamic range of the readout chain can be used for the active signal.
no SBI SBI
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
active SBI (3) active SBI (3) –– resultsresults
no SBI SBIZ10_12P1 vs
Z9_2P2 vs, sbi
01
10100
100015pA
150pA1500pA
0
10
20
30
40
50
60
70
Sta
ndar
dabw
eich
ung
[cm
]
iphot o :iH GL 1:xiphot o [pA]
Standardabweichung Tint=2ms
15pA150pA1500pA
01
10100
100015pA
150pA1500pA
0
10
20
30
40
50
60
70
Sta
ndar
dabw
eich
ung
[cm
]
iphot o :iHGL 1:xiphot o [pA]
Standardabweichung Tint=2ms
15pA150pA1500pA
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Dynamics Measurement (no SBI)Dynamics Measurement (no SBI)
10-3
10-2
10-1
100
101
102
10-3 10-2 10-1 100 101 102 103
Phasedeviation [°] vs Signal and Backlight
Backlightillumination [W/m²]
Sig
nalil
lum
inat
ion
[W/m
²]
0
2
4
6
8
10
12
14
16
18
20
saturation boundaries
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Dynamics Measurement (SBI)Dynamics Measurement (SBI)
10-3
10-2
10-1
100
101
102
10-3 10-2 10-1 100 101 102 103
Phasedeviation [°] vs Signal and Backlight
Backlightillumination [W/m²]
Sig
nalil
lum
inat
ion
[W/m
²]
0
2
4
6
8
10
12
14
16
18
20saturation boundaries
SBI
SBI
Factor of 300 = 50 dB !!!
Factor of 300 = 50 dB !!!
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
SBI SBI -- SummarySummary
LED illumination
Filter
Filter
LED illumination
LED illumination
18‘000 = 85dB
2‘000 = 66dB
400 = 52dB
Filter:-19dB
Burst Operation-14dB
-33dB
SBI: 50dB
LED illumination
Only SBI circuitry allows robust 3d measurement under full sunlight.
Only SBI circuitry allows robust 3d measurement under full sunlight.
Filter + SBI
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Technical challenges and solutionsTechnical challenges and solutions
Technical challenge
high bandwidth
PMD solution
finger structures,short modulation channels
1cm 67ps >100 MHz
high dynamics (sunlight) active SBI, (spectral filter, LED burst)
0.5-5m, r=0.05..1, 10levels 86dBplus sunlight: up to >> Factor 100
01
10100
100015pA
150pA1500pA
0
10
20
30
40
50
60
70
Stan
dard
abw
eich
ung
[cm
]
iphot o :iHGL 1:xiph ot o [pA]
Standardabweichung Tint=2ms
15pA150pA1500pA
01
10100
100015pA
150pA1500pA
0
10
20
30
40
50
60
70
Stan
dard
abw
eich
ung
[cm
]
iphot o :iHGL 1:xip hot o [pA]
Standardabweichung Tint=2ms
15pA150pA1500pA
50 dB
high sensitivity scalable pixel size
typical photo currents down to few 100 fA50% FF
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
ADVANTAGES and USPs
ADVANTAGES and USPs
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Comparison to existing solutionsComparison to existing solutions
Stereo Vision• Shadowing effects• Problems with low contrast scenes• Elaborate computing required• Significant delay times for 3D
calculation• 2 complete cameras PLUS
powerful processor
Laser Scanners• Moving parts• Lots of components• Elaborate mechanics• Expensive• vibration sensitive• slow
PMD is cheaper.PMD has better performance.PMD is smaller in size.
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
AdvantagesAdvantages
Suppression of background illumination (SBI)Various sensor resolutions (pixels scalable in size/sensitivity)
16x16, 32x24, 64x16, 64x48,160x120Highest sensitivityHighest bandwidthHighest contrast
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
PRODUCTSPRODUCTS
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
efectorefector PMDPMD
1D Distance sensor based on PMD principle1D Distance sensor based on PMD principleSensor introduced on HMI 2005Sensor introduced on HMI 2005current pricing (2005): 248,current pricing (2005): 248,-- €€
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Cameras: PMD[vision]®Cameras: Cameras: PMD[visionPMD[vision]]®®
Camera platform: FireWire; Ethernet; SC520 133 MHz CPU, FPGA Spartan II, eLinOS operating systemAvailable Sensors: 64x48 SBI, 64x16 SBI, 160x120 (SBI: Suppression of background illumination)
PMD [vision]®
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
SENSORS: SENSORS: PhotonICsPhotonICs®® PMD PMD
PhotonICs® PMD 1k-S23D Video Sensor Array with Active SBI64x16 Pixels
PhotonICs® PMD 19k3D Video Sensor Array160x120 Pixels, QQVGA
PhotonICs® PMD 3k-S3D Video Sensor Array with Active SBI64x48 Pixels
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
ApplicationsApplications
APPLICATIONSAPPLICATIONS
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Markets and ApplicationsMarkets and Applications
Industrialsafetysurveillanceobject measurementobject recognition…
Industrialsafetysurveillanceobject measurementobject recognition…
Automotivepedestrian safetyACC stop & go (ACC)collision avoidanceparking aidemergency breakman-machine interface (gesture recognition)smart airbag...
Automotivepedestrian safetyACC stop & go (ACC)collision avoidanceparking aidemergency breakman-machine interface (gesture recognition)smart airbag...
RoboticsnavigationAGV ‘scollision avoidance…
RoboticsnavigationAGV ‘scollision avoidance…
Multimediavirtual realityuser interfacesgames video conferencingfilm & televisionproduct presentation
» ...
Multimediavirtual realityuser interfacesgames video conferencingfilm & televisionproduct presentation
» ...
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Background replacementBackground replacement
original data background replaced
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Parking AssistantParking Assistant
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Rear View CameraRear View Camera
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Pedestrian SafetyPedestrian Safety
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Gesture ControlGesture Control
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Keyless DialingKeyless Dialing
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
Advantages of 3D for Gesture RecognitionAdvantages of 3D for Gesture Recognition
Classical 2D-camera, no depth information
Additional information of a 3D-camera
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
3D Siam3D Siam-- ProjektProjekt
3D-Siam 3D-Sensorik für vorausschauende Sicherheitssysteme im Automobil.
BMBF- Projekt (08/2001 bis 07/2005)
Fachhochschule Trier
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
3D3D--Siam Test CarSiam Test Car
IR-Sender
PMD-Kamera
……discovering new dimensionsdiscovering new dimensions© PMDTec GmbH 2005 Dr. R. Lange
THANKS FOR YOUR ATTENTIONTHANKS FOR YOUR ATTENTION
Systematic Investigation of Properties of PMD-Sensors
Systematic Investigation of Properties of PMD-Sensors
1st Range Imaging Research Day, Zürich
Jens Pannekamp, Alexander Steitz
Motivation
• Tremendous success of machine vision(2005: 1 billion € for machine vision in Germany)
• Many real world interactions need 3D!
• Abundant applications in automation & robotics, mobile systems and safety & security
• Compact range sensors become available(PMD Technologies, CSEM, Canesta, 3DV Systems, Matsushita)
Guide for evaluation and selection of 3D-sensors
Applications for 3D-Sensors
• Driver assistence (PMD Technologies)
• Guidance (Claas)
• Collision avoidance (Götting)
• Consumer tracking (Vitracom)
• Bin picking (3 x IPA)
Investigated Range Sensor
• PMD[vision] 1k-S of PMD Technologies
• Sensor array with 64 x 16 pixels
• Suppression of background illumination
• Visualization software CamVis Pro
Data sheet
• Max. range: 7.5 m
• Z-resolution: > 6 mm
• Field of view: 34° (h) x 12° (v)
• Framerate: up to 50 fpsSource: PMD Technologies, Siegen
Example Images
• Upper image:Intensity (grayscale) image of hand in front of chess board
• Lower image:Distance image of same scene – segmentation not disturbed by surface properties
• Problems in practice:Distance values not independent of surface (material, color, texture)
Experimental Setup
• Camera on vertical stage
• 11 materials on plates of size 35 x 35 cm2 :paperaluminium (smooth and sand blasted)steel (smooth and sand blasted)plastics (smooth and rough)ceramic tile (smooth and rough) woodcloth
• Tiltable mount for exchangeable plates
Calibration and Image Evaluation Procedure
• Usage of calibration tool „CamVis Pro“
• Calibration distance: 50 cm
• Calibration material: white paper
• Manual optimization of shutter and integration time for each series of 50 images
• Optimization criterion: Reduction of visible noise in image
• Typical integration time: 10 000 µsec = 0.01 sec
• Manual selection of ROI for relevant pixels, ie. pixels measuring plate
Series of Experiments
a. Influence of distance and materialDistances = {50, 100, 200, 400} cmMaterials = {all materials}50 images for each combination with optimized shutter and integration time
b. Linearity of distance measurementDistances = {50, 100, 200, 400, 550} cmMaterials = {paper, sand blasted aluminium, smooth steel}50 images for each combination with optimized shutter and integration time
c. Estimation of tilt angleTilt anlge = {0°, 5°, 10°, 15°, 30°, 45°, 60°}Distances = {200 cm}Materials = {paper, sand blasted aluminium, smooth steel}50 images for each combination with optimized shutter and integration time
Camera Properties
1. Systematic error for single pixel”Mean that would result from an infinite number of measurements of the same measurand carried out under repeatability conditions minus a true value of the measurand.”
2. Standard uncertainty for single pixel”Uncertainty of the result of a measurement expressed as a standard deviation.”
3. LinearityThe maximum deviation of any points from a straight line drawn as a "best fit"through the point cloud.
4. Estimation of tilt angleTilt angle is extracted from ROI by means of a best fit algorithm and compared to the true value.
Systematic Error: Derivation
For each distance/material combination of measurement series A, distance values of all 50 images and all relevant sensor pixels are averaged. The difference between this averaged value and the true distances is considered as systematic error.
Systematic Error
Systematic Error: Data
Systematic Error [mm] Distance [mm]Material 500,00 1000,00 2000,00 4000,00sand blasted steel 44,87 43,99 169,98 94,76rough ceramic tile 31,80 54,74 187,29 143,75smooth ceramic tile 19,58 42,47 163,66 103,83rough plastics 40,71 41,26 172,95 98,93smooth plastics 28,78 46,95 186,05 92,50smooth steel 9,83 10,48 161,81 117,77smooth aluminium 57,30 62,02 163,30 71,91sand blasted aluminium 7,07 1,36 172,09 194,10wood 36,07 57,91 203,53 142,26paper 3,58 5,25 190,35 196,24cloth 30,59 50,76 188,56 145,99
Systematic Error: Interpretation
• Short distances (50 cm – 100 cm):0.5 cm .. 6 cmDiffuse scattering materials (paper, sand blasted aluminium) perform best
• Longer distances (200 cm – 400 cm):7 cm .. 20 cmSmooth surfaces (smooth aluminium, smooth plastics) perform bestDiffuse scattering surfaces probably do not return enough light (?)
• Material and surface roughness determine quality of distance measurement
• High errors – especially for longer distances – can be attributed partially to calibrationat 50 cm
• Potential for improving (one-point) calibration
Standard Uncertainty: Derivation
For each distance/material combination of measurement series A, the standard deviation is calculated over all 50 images and all relevant sensor pixels.
(Different approach in paper:First, for each distance/material combination of measurement series A an average range image is calculated from the 50 images. Then, the standard deviation is calculated over all relevant sensor pixels.)
Standard Uncertainty
Standard Uncertainty: Data
Standard Uncertainty [mm]Material 500,00 1000,00 2000,00 4000,00sand blasted steel 22,81 19,88 12,82 21,65rough ceramic tile 19,21 18,14 34,24 95,62smooth ceramic tile 27,34 22,49 16,69 31,43rough plastics 21,11 24,94 46,86 128,36smooth plastics 38,26 42,81 74,38 170,09smooth steel 32,46 25,50 22,32 35,00smooth aluminium 115,33 68,32 56,56 30,64sand blasted aluminium 19,75 18,42 23,23 57,32wood 18,33 19,26 21,25 50,36paper 18,72 22,47 26,66 85,33cloth 18,51 20,28 23,45 65,84
Distance [mm]
Standard Uncertainty: Interpretation
• Standard uncertainty for single pixel: 2 cm .. 12 cm
• Plastics performs badly for longer distances (13 cm .. 17 cm)
• For diffuse scatterers (wood, paper, cloth) uncertainty rises from low levels
• Materials with good and near constant performance: sand blasted steel, smooth ceramic tile, smooth steel
• Smooth aluminium with inverse behaviour
• For deeper understanding: measurement of roughness and optical properties
• Standard uncertainty can be reduced by ost-processing (smoothing, median filtering)
Linearity: Derivation
The linearity is derived from plots of measurement series B depicting measured distance values versus true distances. The maximum deviation between any measured distance value and the regression line obtained by a least squares fit is taken as linearity. The distance values are obtained by averaging over all 50 images and all relevant sensor pixels.
Linearity
Linearity: Data
Linearity as Deviationfrom Regression Line [mm]Material 500,00 1000,00 2000,00 4000,00 5500,00sand blasted aluminium 17,30 42,28 77,42 3,86 21,70smooth steel 30,03 24,62 91,47 39,47 2,65paper 10,32 46,89 84,95 23,14 4,60
Distance [mm]
Linearity: Interpretation
• Data shows only little deviations from linear behaviour
• Suggestion for calibration: Removal of systematic errors by means of linear regression
• Remaining non-linear errors 0.3 cm .. 9 cm for measuring range 50 cm .. 550 cm
• Remaining relative errors 1% .. 5%But: This calibration depends on material!
Estimation of Tilt Angle: Derivation
The tilt angle of the plate is estimated from the range images of measurement series C. First, all 50 images of one tilt angle/material combination are averaged. Then, a plane is fitted to the data by means of a least squares fit. The derived orientation of this plane is compared to the true tilt angle.
Estimation of Tilt Angle
Estimation of Tilt Angle: Data
Tilt Angle [°] Reference Angle [°]Material 0,00 5,00 10,00 15,00 30,00 45,00 60,00sand bl.alum. -4,27 1,00 8,22 16,53 40,73 58,19 60,53paper -3,40 1,61 5,88 13,72 32,01 50,32 67,90smooth steel -5,12 4,07 8,33 15,61 42,14 57,39
Estimation of Tilt Angle: Interpretation
• Tilted surfaces can be measured up to at least 60°
• Paper shows a near linear behaviour
• Number of available pixels for plane fit decrease with tilt angle
ConclusionSystematic error and standard uncertaintySystematic error and standard uncertaintySystematic error and standard uncertaintySystematic error and standard uncertainty•••• For short distances (50 cm, 100 cm) diffuse scattering materials such as paper or sand
blasted aluminium perform best (systematic error 0,4 cm resp. 0,7 cm).•••• For longer distances (200 cm, 400 cm) smooth surfaces such as smooth aluminium or
smooth plastics yield best results (systematic error 7 cm resp. 9 cm).•••• Standard uncertainty ranges from 1 cm to 17 cm and depends strongly on material and
distance
LinearityLinearityLinearityLinearity•••• If systematic errors are removed by means of a linear regression, the remaining non-
linear errors are in the range of 0,3 cm to 9 cm for a measuring range of 50 cm to 550 cm. This corresponds to relative errors ranging from less than 1% up to 5%.
Tilted surfacesTilted surfacesTilted surfacesTilted surfaces•••• Tilted surfaces can be measured up to at least 60°.
Open Questions• Investigation how systematic error and uncertainty depend on surface parameters
(roughness, optical properties, color etc.)
• Comparison of cameras of different manufacturers
Acknowledgement
The technical support of PMD Technologies is strongly appreciated.
Research Project „SmartVision“
• Project in German InnoNet-Program
• Start: January 2006
Objectives
• Development of algorithms for object tracking and object recognition(IWR, IPA, 3Soft)
• Applications in the fields of automation & robotics, mobile systems and safety & security
Unified analysis of the performanceand physical limitations of opticalrange-imaging techniques
Peter SeitzVice President, CSEM SA, ZurichProfessor of Optoelectronics, University of Neuchâtel
Range Imaging Days, ETH Zurich, September 8-9, 2005
Peter Seitz :: 05.10.2005 :: Page 2
Content
§ Introduction and motivation
§ Taxonomy of optical range imaging techniques
§ Limitations of the generation and detection of light
§ Linear shift-invariant (LSI) systems
§ A unified framework for optical RIM system analysis
§ Quantum noise limited phase measurement
§ Comparison of RIM system performance limitations
§ Conclusions
§ Summary
Peter Seitz :: 05.10.2005 :: Page 3
Peter Seitz :: 05.10.2005 :: Page 4
Peter Seitz :: 05.10.2005 :: Page 5
Taxonomy of optical range imaging techniques
Interferometry Thomas Young (1801)
lens
light source
rotatingcogwheel
observer
beam splitter planemirror
lens
lens
lens
lens
light source
rotatingcogwheel
observer
beam splitter planemirror
lens
lens
lens
Time-of-flight ranging Armand Fizeau (1849)
Triangulation Egyptian surveyors (1400 B.C.)
Peter Seitz :: 05.10.2005 :: Page 6
Optical setups of the basic ranging methods
x
α
z
Triangulation
z
Interferometry
z
Time-of-flight
Peter Seitz :: 05.10.2005 :: Page 7
0.00
0.02
0.04
0.06
0.08
0.10
0.12
0.14
0 5 10 15 20 25
Photon number
Pro
babi
lity
Poisson
Bose-Einstein
n=10
Limitations of light generation and detection (I)
Generation and detection of light are statistical, noisy processes: - Coherent light (independent random events): Poisson statistics - Thermal light (correlated events): Bose-Einstein statistics
NPoisson =σ
2NN
EinsteinBose
+
=−σ
Peter Seitz :: 05.10.2005 :: Page 8
1.E+00
1.E+02
1.E+04
1.E+06
1.E+08
1.E+10
1.E+00 1.E+02 1.E+04 1.E+06 1.E+08 1.E+10
Number of photons
Sta
nd
ard
dev
iati
on
Poisson Bose-Einstein
Limitations of light generation and detection (II)
Peter Seitz :: 05.10.2005 :: Page 9
Linear shift invariant (LSI) systems
LSI
In the complex function space, harmonic functions are eigenfunctions of all LSI systems. The function system of the harmonic functions is complete in the linear space of square integrable functions
Peter Seitz :: 05.10.2005 :: Page 10
Unified framework for optical RIM system analysis
x
z
x
P
z
P
t
Pλ/2 1/fΛ
α
z
z
πϕ
α 2tanTRIz
Λ=
πϕλ22
INTz =π
ϕ22TOF
fc
z =
Peter Seitz :: 05.10.2005 :: Page 11
Quantum noise limited phase measurement (I)
s
F(s)
B
A
ϕ
S=1/f
∆s
Fi
−−
=
+++=
−+−=
20
13
3210
220
213
arctan
4
2)()(
FFFF
FFFFb
FFFFa
ϕ
Ss
sb
Bs
aA
∆=
∆=
∆=
πδ
δδ
sin
Peter Seitz :: 05.10.2005 :: Page 12
s
F(s)
B
A
ϕ
S=1/f
∆s
Fi
Quantum noise limited phase measurement (II)
ab
sF k
k k 2
23
0
=
∂∂
= ∑=
ϕσϕ
variances si = Fi
Peter Seitz :: 05.10.2005 :: Page 13
Comparison of RIM system performance limitations
πσ
ασ ϕ
2tan,Λ
=TRIz π
σλσ ϕ
22, =INTz π
σσ ϕ
22, fc
TOFz =
l
b
αΛ
z
xx
bbz
l==
αtan
Triangulation
z
Interferometry Time-of-flight
Peter Seitz :: 05.10.2005 :: Page 14
Extending the non-ambiguity range
21
2112 Λ−Λ
ΛΛ=Λ
The non-ambiguity range is not restricted to half of the wavelength: By using two different wavelengths Λ1 and Λ2 a synthetic wavelength Λ12 is produced, leading to a much larger non-ambiguity range
Peter Seitz :: 05.10.2005 :: Page 15
Conclusions and Summary
§ Photon quantum noise is one of the key factors limiting the performance
of optical range imaging methods (diffraction, speckle, etc.)
§ Measurement = Comparison with known common standard. Conclusion:
Three fundamental methods: Triangulation, interferometry, time-of-flight
§ Unified analysis of all optical range imaging methods in the framework of
LSI systems: Identical functional relationship – phase measurement.
§ Phase measurement precision limited by photon quantum noise
§ Largely unexplored notion: Combination of various methods
§ The different methods have more in common than one might think.
Conclusion: Share approaches, insights and ideas – work together !
T h a n k y o u f o r y o u r a t t e n t i o n.
CCD/CMOS Lock-In Pixel for Range ImagingChallenges, Limitations and State-of-the-Art
Bernhard BuettgenCSEM, Centre Suisse d’Electronique et de Microtechnique SA
September 8, 2005
Overview
Phase measurement for range imaging
Shot-noise limitation
Pixel concepts
Distance accuracy
Dynamic range
Background-light suppression
Enhanced pixel structures
Summary
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 1
Homodyne phase measurementIntensity Modulation:
Pulse: direct ‘’time-of-flight’’
Continuous Wave: phase delay between emitted and detected signal
Simultaneous acquisition of depth information and brightness
pixelmatrixL
objects
Popt
illumination
sensor readout andimage display
start
stop
ϕπ
⋅⋅
=modf
cL4
c: speed of light
fmod: modulation frequency
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 2
Pixel-Inherent Sampling
Emitted signal
Detected signal
Measurements: four sampling points A0 ... A3
)2sin()( 00 tfPPtP modopt π⋅+=
)2sin()( ϕπ +⋅+= tfABtN modmeasmeasel
t
Nel
A0
A1
A2
A3
Popt
emitted signal
detected signal
ϕ Bmeas
Ameas
P0
⎟⎟⎠
⎞⎜⎜⎝
⎛−−
=31
20arctanAAAA
ϕ
Brightness information
Accuracy information
Distance information
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 3
Figure of Merit: Demodulation Contrast
sig
meas
AA
c =demod
Measured signal amplitude [#e]
Signal offset [#e]
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 4
Physical Limitation by Photon Shot Noise
sigL Ac
Bfc
⋅⋅
⋅⋅=
demodmod 24πσ
t
Nel
A0
A1
A2
A3
Popt
emitted signal
detected signal
ϕ Bmeas
Ameas
P0
c: light velocity [m/s]
fmod: modulation frequency [Hz]
cdemod: demodulation contrast [-]
B: intensity [#e]
A: amplitude [#e]
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 5
Pixel Concepts
1-Tap 2-Tap 4-Tap N-Tap
Exploitation of optical energy
50% lost 100% used 100% used 100% used
Mismatch problems No Yes Yes Yes
Fill-factor +++ ++ - ---
Number of acquisitions 4 2 1 1
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 6
2-Tap Pixel
Two samples per
acquisition
CCD-arrangement for
sampling the light
CMOS read-out amplifiers
Charge transport mainly
based on diffusion
processes
Buried channel enhances
the performance
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 7
Challenge I: Distance Accuracy
Precise 3D measurements in real-time without …increasing the total emitted optical power (keeping eye safety)increasing the optics aperture (keeping the systems small)using electronics with better noise performance(keeping the cost level low)
Goal: Std.=1mm or even below!
Enabling new applicationsBiometricsIndustrial packaging
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 8
Accuracy – Limitationssig
L AcB
fc
⋅⋅
⋅⋅=
demodmod 24πσ
104 105 1060
0.5
1
1.5
2
2.5
3
mean number of photo-generated electrons per sample [-]
accu
racy
[cm
]
fmodsweep from 10MHzto 80MHz
Assumptions:
- samples shifted by 90 degrees
- demodulation contrast of 64%
- no delays in the charge transport
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 9
Accuracy – Modulation Frequency
Charge transport dominated by …
… drift processes … diffusion processes
Ultdrift µ
2
=Dltdiff
2
=
110 3V U:Exp. ≈⇒=drift
diff
tt
Higher cut-off frequencies using drift fields!
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 10
Challenge II: Dynamic Range
3D-images are composed of objects at
short distances with low and high reflection coefficients
long distances with low and high reflection coefficients
Pixel capability of measuring all distance between the longest and shortest
distances
No saturation!
Minimum standard deviation of the distance measurement!
sigL Ac
Bfc
⋅⋅
⋅⋅=
demodmod 24πσ
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 11
Dynamic Range – Limitations
Dynamic range sets the minimal and maximal storable amount of
charges into relation!
Minimum amount of charges
defines the smallest uncertainty of the distance measurement
Maximum amount of charges
defined by the brightest points in the scenery
[ ]⎥⎥⎦
⎤
⎢⎢⎣
⎡⋅⎟⎟
⎠
⎞⎜⎜⎝
⎛⋅=
min
max
2
min
max10log20δδ
RRdBDR
sigL Ac
Bfc
⋅⋅
⋅⋅=
demodmod 24πσ
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 12
Dynamic Range – Example
Distance range: Rmin = 20cm, Rmax = 5m
Reflection coefficients: δ = 5%, δ = 100%
DR = 82dB (Factor of 12500)
Minimum accuracy of 1cm requires 60.000 electrons
Also 750.000.000 electrons must be storable!
sigL Ac
Bfc
⋅⋅
⋅⋅=
demodmod 24πσ
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 13
Challenge III: Background Light
400 500 600 700 800 900 1000 11000
0.5
1
1.5
wavelength [nm]
optic
al p
ower
den
sity
[kW
/m2 / µ
m]
bandwidth
~800kW/m2/µm
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 14
Background-Light Suppression – Limitations
0 5 10 15 200
2
4
6
8
background to signal ratio [-]
accu
racy
[cm
]
signal electrons:1) 100002) 316003) 1000004) 3160005) 1000000
signal power 1
2
34
5
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 15
Enhanced Pixel for High Dynamic Range
Pixel-wise integration (PWI)
Functionality verified
1 1.5 2 2.5 3 3.50
2
4
6
8
real distance [m]
mea
sure
d di
stan
ce [m
] normal modepwi mode
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 16
Enhanced Pixel for High Background Light
Difference of charge packets is processed in-pixel
0 20 40 60 80 100 120 1400
2
4
6
8
10
12
14
ratio background to signal [-]
accu
racy
[cm
]
theorymeasurement
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 17
Summary and Outlook
Smart pixels for fast optical distance measurement based on TOFChallenges for future designs
High accuracy high modulation frequenciesHigh dynamic range adaptive integration timeBackground-light suppression subtraction of charge packets
State-of-the-artEnhanced pixel structures for extended dynamic range and background light suppressionAspects of high modulation frequencies to be investigated!
New pixel architectures are required that enable the charge transport by electric drift fields.
Bernhard Buettgen :: RIM Day 2005 :: ETH Zurich 18
T h a n k y o u f o r y o u r a t t e n t i o n.