foundations of time-of-flight cameras and their … · camcube ® pmd technologies ... •...
TRANSCRIPT
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery
Foundations of Time-of-Flight Cameras and their
Application to Surface Reconstruction
Andreas Kolb
Computer Graphics Group, Institute for Vision and Graphics
University of Siegen, Germany
Overview
• Time-of-Flight Cameras: The Photonic Mixing Device (PMD)
Principle
• PMD Characteristics and Calibration
• 2D/3D Sensor Fusion
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• Endoscopic Application
• Summary
Overview
• Time-of-Flight Cameras: The Photonic Mixing Device (PMD)
Principle
• PMD Characteristics and Calibration
• 2D/3D Sensor Fusion
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• Endoscopic Application
• Summary
Time of Flight Cameras
• At a glance: Cameras …
o … are active camera-like distance sensors
o … deliver range maps plus intensity values instantaneously (up to 40 FPS)
o … have moderate resolution (max. 204² px)
• Basic Technological Approaches for ToF-Cameras
o Photonic Mixing: Electrical, on-chip mixing in smart pixels
o Optical Shuttering: Optical mixing
• Current Sensors
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• Current Sensors
CamCube®
PMD TechnologiesSwissRanger 4000®
Mesa ImagingZCam®
3DVSystems
Intensity Modulation Approach
• Photonic Mixing Device (PMD):
o Active time-of-flight approach
o Intensity modulation of incoherent light (NIR)
o Correlation of reference signal and optical signal
• “Smart pixel” realizes a photo mixing detector to sample the
correlation function with Standard CMOS-technology
• Suppression of background intensity
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• Correlation Function
o Reference signal: ( internal phase-shift)
o Optical signal:
o Correlation:
o Sinusoidal signal: Trigonometric calculus yields
Intensity Modulation Approach
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
o Sinusoidal signal: Trigonometric calculus yields
Intensity Modulation Approach
• Distance computation
o Sampling for 4 different internal phase shifts yields four phase
images
o Demodulation gets phase delay
o Applying speed of light gets distance
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
o Applying speed of light gets distance
[Lange00]
Overview
• Time-of-Flight Cameras: The Photonic Mixing Device (PMD)
Principle
• PMD Characteristics and Calibration
• 2D/3D Sensor Fusion
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• Endoscopic Application
• Summary
Sensor Characteristics & Challenges
• Low Resolution compared to 2D sensors � “Flying Pixels”
• Systematic Error (wiggling): No ideal sine-signal
� demodulation yields wrong distance
• Intensity-related Error: Low incident active light
o yields increased noise level and
o non-zero mean shift in distance measurement
• Motion artifacts due to multi-exposure approach
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
Background plane
Foreground boxmoving to the left
Flying pixel
Motion artifacts
Calibration
• Classical intrinsic and extrinsic calibration based on intensity
images using e.g. OpenCV
• Depth-Calibration requires reference (depth) data
o Track-line, robot (costs!)
o Standard image based techniques for extrinsic parameters (plane
estimation)
• Purely image based techniques works only for “high res”, i.e. ≥
120x160 px
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
120x160 px
• Alternative: Calibration based on several sensors
Distorted PMD-Image Undistorted PMD-Image
• Observation:
o Periodical distance error with varying periodicity and amplitude
• Correction functions used
o Lookup tables , e.g. [Kahlmann06]
o B-splines, e.g. [Lindner06]
600
derivationbspline
600mm
Systematic Error (Wiggling)
60
derivationadjustet
80mm
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
Depth error with fitted B-spline
-600
-400
-200
0
200
400
0 1000 2000 3000 4000 5000 6000 7000 8000
bspline
-600mm -140
-120
-100
-80
-60
-40
-20
0
20
40
4000 4500 5000 5500 6000 6500 7000 7500
adjustet
-120mm
Illumination-Based Error
• Observation: Nonlinear dependency from illumination
• Approaches to considering this error:
o Combined depth & illumination calibration using bi-variate B-splines
�requires a large amount of reference data [Lindner08]
o Separation of depth and intensity error � less amount of reference data
0mm 0
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
Deviation between measured and real distance over intensity with fitted B-splines-200mm-200
-150
-100
-50
0 500 1000 1500 2000
Illumination-Based Error (Cont‘d)
• Approach: Separate systematic and intensity error correction
• Problem: Intensity relative error partially corrected by
systematic correction � Intensity dependent correction need to correct wiggling offset
• Next ToF-cam generation may solve the problem electronically
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
Remaining distance error after wiggling correction (Ø 1.62cm), coupled (Ø 0.97cm) and decoupled (Ø 0.98 cm) intensity correction [Lindner10]
Motion Compensation
• Observation: Subsequent phase images yield motion artifacts
• Approach: Align phase images Ai using optical flow [Lindner09]
• Problems to be solved:
o PMD pixels have inhomogeneous gain behavior
o Mixing & inhomogeneous illumination � non-comparable intensities
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
Correction formixing
Correction forradial falloff
Motion Compensation
• Results:
o Robust and accurate correction for motion artifacts
o Performance: 10-20 FPS on a current graphics hardware
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
Reference w/o motion comp. with motion comp.
Overview
• Time-of-Flight Cameras: The Photonic Mixing Device (PMD)
Principle
• PMD Characteristics and Calibration
• 2D/3D Sensor Fusion
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• Endoscopic Application
• Summary
• Goal: Combine ToF-camera with standard RGB-camera
o Compensate for low ToF-resolution
o Add additional colour information
• General fusion approaches
o Binocular, i.e. two separate cameras � two optical centers
o Monocular, e.g. using a beam-splitter to separate NIR and VIS
2D/3D Sensor Fusion
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• Calibration
o In general: Multi-camera 2D/3D fusion approaches are more robust
o Tool download under: www.mip.informatik.uni-kiel.de
Image from [Ghobadi10]
2D/3D Sensor Fusion: Some General Issues
• Fusing Low Resolution ToF with High Resolution RGB Sensor
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• Binocular approach leads to false colour assignment (near field)
Left: Per-pixel colour assignment; Right: Texture projection
False colorassignment
RGB sensor
PMD sensor
False color assignment below hand
Overview
• Time-of-Flight Cameras: The Photonic Mixing Device (PMD)
Principle
• PMD Characteristics and Calibration
• 2D/3D Sensor Fusion
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• Endoscopic Application
• Summary
3D ToF Endoscopy: System Setup
• Main Advantages of ToF-Cameras:
o (Almost) zero computational effort to get range images
o Compact devise without any baseline requirement
o Active sensing � detection of unstructured surfaces
• General technical concept
o Coupling of ToF-illumination with the endoscope’s light transport system
o Monocular approach: Beam splitter allows for 2D/3D data acquisition
• First Prototype [Penne09]
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• First Prototype [Penne09]
o Single fiber-coupled high-power laser diode replaces the illumination unit
o Low resolution PMD-chip (48x64) only (no additional 2D camera)
Image from [Winter09]
3D ToF Endoscopy: Data Processing
• General Goal & Challenge:
o Calibration: Standard ToF-calibration plus correction for endoscope length
o High-speed processing of ToF data (204² px * 20FPS = 800k points per
sec)
o Incremental surface reconstruction requires new algorithms
• Classical Surface Reconstruction:
o Successive steps for acquisition,
pre-processing, mesh
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
Image from [Winter09]
pre-processing, mesh
generation and post-
processing
• Incremental Surface Reconstruction requires
o Parallel processing of new points and triangle in each step
o Advancing front approach: Ongoing finalization of geometry � finalized portions can not be altered if new data comes in
o Ongoing refinement: Revisiting object regions � new points influence the final surface
3D ToF Endoscopy: Ridge-Based Approach [Winter09]
• General idea of the ridge-based approach
o Each 3D point distributes “volume” in 3-space using a kernel function
o Superposition of all points give a density-like scalar field (stored in a grid)
o Final surface is found by following the ridge of the scalar field
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
• Incremental version
o Ongoing integration of point contributions into the scalar field
o Reconstruction when some local criterion is met
Image from [Winter09]
Image from [Winter09]
3D ToF Endoscopy: Results [Winter09]
• [Winter09] evaluated his work using an silicon liver
o Ideal data based on a 3D scan of the model
o Real ToF-data (48x64 pixel)
Liver model
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
Real ToF data
Ideal range data
Summary
• Time-of-Flight Technology is very promising for laparoscopic
applications
o Compact sensor principle (no baseline) for real-time range maps
• Advantages and Achievements
o Active technology � even weakly textured surfaces can be handled
o Various calibration issues (wiggling, intensity related error, motion
artefacts) have already been solved
o Low resolution & missing colour can be overcome with sensor fusion
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
o Low resolution & missing colour can be overcome with sensor fusion
• Further Challenges for Laparoscopic Applications
o Hardware integration is quite tedious
o Real-time data processing, e.g. using the data redundancy is application
dependant
Thank you for your attentionThank you for your attention
References
[Ghobadi10] Ghobadi S.: Real time object recognition and tracking using 2D/3D images . PhD thesis, University of Siegen, 2000.
[Kahlmann06] Kahlmann T., Remondino F., Ingensand H.: Calibration for increased accuracy of the range imaging camera SwissRanger. In Image Eng. & Vision Metrology (2006).
[Lange00] Lange R.: 3D Time-Of-Flight Distance Measurement with Custom Solid-State Image Sensors in CMOS/CCD-Technology. PhD thesis, University of Siegen, 2000.
[Lindner06] Lindner M., Kolb A.: Lateral and depth calibration of PMD-distance sensors. In Intl. Symposium on Visual Computing (2006), vol. 2 of LNCS, Springer, pp. 524–533.
[Lindner07] Lindner M., Kolb A.: Calibration of the intensity-related distance error of the PMD TOF-camera. In Intelligent Robots and Computer Vision XXV (2007), vol. 6764,
A. Kolb – Time-of-Flight Cameras
Tutorial on 3D Surface Reconstruction in Laparoscopic Surgery.22nd September ,Toronto, Canada
PMD TOF-camera. In Intelligent Robots and Computer Vision XXV (2007), vol. 6764, SPIE.
[Lindner09] Lindner M., Kolb A.: Compensation of motion artifacts for time-of-flight cameras. In Dynamic 3D Imaging (2009), vol. 5742 of LNCS, Springer, pp. 16 – 27.
[Lindner10] Lindner M.: Calibration and Real-Time Processing of Time-of-Flight Range Data. PhD thesis, University of Siegen, 2010.
[Penne09] Penne J., Höller K., Stürmer M., Schrauder T., Schneider A., Engelbrecht R., Feußner H., Schmauss B., Hornegger J.: Time-of-Flight 3-D Endoscopy. MICCAI, Springer LNCS, Volume 5761, 467-474, 2009.
[Winter09] Winter M.: Image-based Incremental Reconstruction, Rendering and Augmented Visualization of Surfaces for Endoscopic Surgery. PhD thesis, University of Erlangen, 2009.