exposure fusion for time-of-flight imaging
DESCRIPTION
Exposure Fusion for Time-Of-Flight Imaging. Uwe Hahne, Marc Alexa. Depth imag ing. [Audi]. Applications. [Shotton2011]. Depth imaging. [ PMDTec ]. [Microsoft]. Devices. [ PTGrey ]. General problem. Video #1. Our results. Video #2. Contribution. + No calibration - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/1.jpg)
PG 2011Pacific Graphics 2011
The
19th
Pac
ific
Conf
eren
ce o
n Co
mpu
ter G
raph
ics a
nd A
pplic
ation
s (P
acifi
c Gr
aphi
cs 2
011)
will
be
held
on
Sept
embe
r 21
to 2
3, 2
011
in K
aohs
iung
, Tai
wan
.
PG 2011Pacific Graphics 2011
The
19th
Pac
ific
Conf
eren
ce o
n Co
mpu
ter G
raph
ics a
nd A
pplic
ation
s (P
acifi
c Gr
aphi
cs 2
011)
will
be
held
on
Sept
embe
r 21
to 2
3, 2
011
in K
aohs
iung
, Tai
wan
.
PG 2011Pacific Graphics 2011
The
19th
Pac
ific
Conf
eren
ce o
n Co
mpu
ter G
raph
ics a
nd A
pplic
ation
s (P
acifi
c Gr
aphi
cs 2
011)
will
be
held
on
Sept
embe
r 21
to 2
3, 2
011
in K
aohs
iung
, Tai
wan
.
Exposure Fusion for Time-Of-Flight ImagingUwe Hahne, Marc Alexa
![Page 2: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/2.jpg)
Depth imaging
[Shotton2011]
[Audi]
Applications
![Page 3: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/3.jpg)
[PMDTec][Microsoft]
[PTGrey]
Devices
Depth imaging
![Page 4: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/4.jpg)
General problem
Video #1
![Page 5: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/5.jpg)
Our results
Video #2
![Page 6: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/6.jpg)
Contribution
+ No calibration+ Real-time capable+ Error reduction
- Only for time-of-flight imaging...
![Page 7: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/7.jpg)
[PMDTec]
![Page 8: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/8.jpg)
[PMDTec]
Popt
t
![Page 9: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/9.jpg)
[PMDTec]
Popt
t
50 ns (ω = 20 MHz)
Phase shift distanceAmplitude intensity
![Page 10: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/10.jpg)
[PMDTec]
Integration time
Popt
t
between 50 and 3000 µs
![Page 11: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/11.jpg)
Problem
Noisy background
Noisy foreground
Short integration time (50 µs) Long integration time (1250 µs)
![Page 12: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/12.jpg)
Sampling
S0 S1 S0S2 S3
Popt
t
![Page 13: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/13.jpg)
Sampling
S0 S1 S0S2 S3
Popt
t
Short integration time low SNR
![Page 14: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/14.jpg)
Sampling
S0 S1 S0S2 S3
Popt
t
Long integration time saturation
![Page 15: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/15.jpg)
What is the optimal integration time?
![Page 16: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/16.jpg)
1. Adaptation (May2006) Updating the integration
time based on data from previous frames.
Only a global solution
Existing approaches
[Lindner2007]
2. Calibration (Lindner, Schiller, Kolb,...) Capture samples of known geometry
and correct the measurements. Laborious and device specific
Video #3
![Page 17: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/17.jpg)
[Lindner2007]
1. Adaptation (May2006) Updating the integration time based
on data from previous frames. Only a global solution
2. Calibration (Lindner, Schiller, Kolb,...) Capture samples of known
geometry and correct the measurements.
Laborious and device specific
Existing approaches
![Page 18: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/18.jpg)
Our approach
How is this problem solved in photography?
High Dynamic Range (HDR) Imaging
![Page 19: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/19.jpg)
HDR imaging
Over-exposed
Details
Under-exposed
Details
Missing details
[Images are courtesy of Jacques Joffre]
Radiance map 101011101001010110101010101010010010101001001011000010100101000101001101011101001010110101010101010010010101011101001010110101010101010010
![Page 20: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/20.jpg)
Over-exposed
Under-exposed
Depth imaging
![Page 21: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/21.jpg)
Algorithm
1. Capture a series of depth images with varying integration times.
2. Compute weights using quality measures.
3. Fuse images together as affine combination of weighted depth maps.
![Page 22: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/22.jpg)
Depth maps
Process overviewLaplace pyramids Weight maps
Fused pyramid Resulting fusion
![Page 23: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/23.jpg)
Depth maps
Process overviewLaplace pyramids Weight maps
Fused pyramid Resulting fusion
![Page 24: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/24.jpg)
Quality measuresContrastWell exposedness Entropy Surface Weight maps
![Page 25: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/25.jpg)
Quality measuresWell exposedness
Emphasizes those pixels where the amplitude is in a “well” range and hence removes over- and underexposure.
[Mertens2007]
![Page 26: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/26.jpg)
Quality measuresContrast
Good contrast in the amplitude image indicates absence of flying pixels.
[Mertens2007]
![Page 27: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/27.jpg)
Quality measuresEntropy
The entropy is a measure for the amount of information.
[Goshtasby2005]
![Page 28: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/28.jpg)
Quality measuresSurface
[Malpica2009]
A measure for surface smoothness which indicates less noise.
σ = Gaussian blurred squared depth mapμ = Gaussian blurred depth map
![Page 29: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/29.jpg)
Quality measuresContrastWell exposedness Entropy Surface Weight maps
![Page 30: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/30.jpg)
Depth maps
Process overviewLaplace pyramids Weight maps
Fused pyramid Resulting fusion
![Page 31: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/31.jpg)
Evaluation
Depth map quality• What is the reference?
![Page 32: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/32.jpg)
Reference
We compared our results to single exposures with the integration time t’.
The integration time t’ is the global optimum found by the method from [May2006].
We use N = 4 exposures (i = 0..N-1) with integration times .
![Page 33: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/33.jpg)
Evaluation
Depth map quality• What is the reference?• What does quality mean?
- Less temporal noise Stability over time test
![Page 34: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/34.jpg)
Capture a static image over time and determine the standard deviation for each pixel.
Overall reduction of mean standard deviation by 25%.
Stability over time
Comparison
F > S
F < S
F ≈ S
mmSingle exposure (S)Fused result (F)
![Page 35: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/35.jpg)
Evaluation
Depth map quality• What is the reference?• What does quality mean?
- Less temporal noise Stability over time test- Better processing results Run ICP and compute
3D reconstruction error
![Page 36: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/36.jpg)
3D reconstruction
![Page 37: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/37.jpg)
Evaluation
Depth map quality What does quality mean?
Less temporal noise Stability over time test Better processing results Run ICP and
compute 3D reconstruction error Impact of each quality measure
Computation times
![Page 38: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/38.jpg)
Computation
![Page 39: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/39.jpg)
Evaluation
Depth map quality What does quality mean?
Less temporal noise Stability over time test Better processing results Run ICP and
compute 3D reconstruction error Impact of each quality measure
Computation times Precision Plane fit in planar regions
![Page 40: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/40.jpg)
We captured test scenes with planar regions.
Planar regions
![Page 41: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/41.jpg)
We ran our fusion algorithm ...
Planar regions
Depth maps Laplace pyramids Weight maps
Fused pyramid Resulting fusion
![Page 42: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/42.jpg)
Planar regions
ContrastWell expo. Entropy Surface Weight maps
...and compared the fusion results of all combinations of quality measures.
![Page 43: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/43.jpg)
Planar regions
We identified planar regions...
![Page 44: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/44.jpg)
Planar regions
...and fitted planes into the 3D data.
![Page 45: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/45.jpg)
Planar regions
![Page 46: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/46.jpg)
1 cm
Planar regions
Surface
Only contrast does work alone
![Page 47: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/47.jpg)
Planar regions
Combinations are valuable
Combinations are valuable
Surface
![Page 48: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/48.jpg)
Planar regions
All four quality measures lead to the smallest error
Surface
![Page 49: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/49.jpg)
Conclusion
A new method for time-of-flight imaging:+ No calibration+ Real-time capable+ Reduced error
![Page 50: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/50.jpg)
Kaohsiung, Taiwan
Pacific Graphics 2011.
52 |
Kaohsiung, Taiwan
Pacific Graphics 2011.
52 |
Kaohsiung, Taiwan
Pacific Graphics 2011.
52 |
Thank you
Acknowledgements- Martin Profittlich (PMDTec) for TOF camera
support.- Bernd Bickel and Tim Weyrich for helpful
discussions.
![Page 51: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/51.jpg)
PG 2011Pacific Graphics 2011
The
19th
Pac
ific
Conf
eren
ce o
n Co
mpu
ter G
raph
ics a
nd A
pplic
ation
s (P
acifi
c Gr
aphi
cs 2
011)
will
be
held
on
Sept
embe
r 21
to 2
3, 2
011
in K
aohs
iung
, Tai
wan
.
PG 2011Pacific Graphics 2011
The
19th
Pac
ific
Conf
eren
ce o
n Co
mpu
ter G
raph
ics a
nd A
pplic
ation
s (P
acifi
c Gr
aphi
cs 2
011)
will
be
held
on
Sept
embe
r 21
to 2
3, 2
011
in K
aohs
iung
, Tai
wan
.
PG 2011Pacific Graphics 2011
The
19th
Pac
ific
Conf
eren
ce o
n Co
mpu
ter G
raph
ics a
nd A
pplic
ation
s (P
acifi
c Gr
aphi
cs 2
011)
will
be
held
on
Sept
embe
r 21
to 2
3, 2
011
in K
aohs
iung
, Tai
wan
.
Exposure Fusion for Time-Of-Flight ImagingUwe Hahne, Marc Alexa
![Page 52: Exposure Fusion for Time-Of-Flight Imaging](https://reader036.vdocuments.net/reader036/viewer/2022062521/56816956550346895de10458/html5/thumbnails/52.jpg)
ReferencesShotton, Jamie, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. “Real-Time Human Pose Recognition in Parts from Single Depth Images.” In CVPR, (to appear), 2011.
Büttgen, Bernhard, Thierry Oggier, Michael Lehmann, Rolf Kaufmann, and Felix Lustenberger. CCD / CMOS Lock-In Pixel for Range Imaging : Challenges , Limitations and State-of-the-Art. Measurement, 2005.
Lindner, Marvin, and Andreas Kolb. “Calibration of the intensity-related distance error of the PMD TOF-camera.” Proceedings of SPIE (2007): 67640W-67640W-8.
Goshtasby, A. “Fusion of multi-exposure images” Image and Vision Computing (2005),23 (6),p. 611-618
Mertens, T., Kautz, J., Van Reeth, F. “Exposure Fusion” 15th Pacific Conference on Computer Graphics and Applications (PG'07), p. 382-390
Stefan May, Bjorn Werner, Hartmut Surmann, Kai Pervolz “3D time-of-flight cameras for mobile robotics”, IEEE/RSJ International Conference on Intelligent Robots and Systems 2006, p. 790-795