signal processing and analysis - miun.se
TRANSCRIPT
W
V
Benny Thörnberg
Associate professor
in electronics
Signal Processing and Analysis3D Image Imaging
Copyright (c) Benny Thörnberg 1:44
Outline
•Scanning shadow shape curves
•Coded light
•Laser triangulation
•Stereo Imaging
•Time Of Flight Imaging
Copyright (c) Benny Thörnberg 2:44
Acquiring shadow shapes using area
scan imaging Illumination lamp
Hi-Light Screen Camera
Silhouette image of frog
1.S. Pansang and C. Kimpan, “3-D object recognition from shadow shape curves”,
Proceedings from 2000 IEEE Asia-Pacific Conference on Circuits and Systems.
Electronic Communication Systems, pp. 437-40, Tianjin China, 2000
Copyright (c) Benny Thörnberg 3:44
3D image acquisition using shadows
Camera
Illumination
Shadow
Copyright (c) Benny Thörnberg 4:44
3D image acquisition using coded light
Projector
Camera
Stripe
patterns
Every point on the surface will be illuminated by a sequence
of light-no-light that is unique for a certain angle of the
projected light.
1.B. Achermann, X. Jiang and H. Bunke, “Face recognition using range
images”, Proceedings. International Conference on Virtual Systems and
MultiMedia VSMM '97, pp. 129-36, Geneva Switzerland, 1997.
Copyright (c) Benny Thörnberg 5:44
2D image acquisition using line scan
Reference: C. Steger, M. Ulrich and C. Wiedermann, Machine Vision Algorithms and Applications
A 1D image sensor can be set up to acquire
a 2D projection of a 3D scene. Motion is then used to capture the second dimension
of the projection.
Copyright (c) Benny Thörnberg 6:44
3D image acquisition using laser
triangulationA 3D representation of the 3D
scene is acquired from using
triangulation with structured
light.
R
X
Y
Sheet of light laser 2-D image sensor
d Scanning in
X-dimension An area scan sensor is used
in combination with motion to
capture the third image
dimension.
Light intensity information can be simultaneously acquired.
Copyright (c) Benny Thörnberg 7:44
Laser triangulation
Laser
H r
Object
Range reference level β
F
-D
D
d
α
rMAX
Projection center of lens
Principal distance
of lens
Φ
S
Copyright (c) Benny Thörnberg 8:44
Laser triangulation
Laser
H r
Object
Range reference level β
F
-D
D
d
α
rMAX
Projection center point of lens
Principle distance
of lens
Φ
S
20
)tan()tan(
)tan()tan()tan( πα
α
αα<<∧
+Φ
⋅Φ−=
SHr
20
)tan(
)tan()tan(
πβ
β
β<<∧
+
−=Φ
dF
dF
20
)tan(
))tan(( πβ
β
β<<∧
+
−=
DF
DFHS
Copyright (c) Benny Thörnberg 9:44
Laser triangulation
Laser
H r
Object
Range reference level β
F
-D
D
d
α
rMAX
Focal point of lens
Principle distance
of lens
Φ
S
If both the laser and the camera
angle α and β equal 45 degrees, a perfect linearity and
highest SNR will be achieved
while minimizing the risk of
occlusion.
F
SHdSHF
F
dFSdFHr
2
)()(
2
)()( ++−=
−−+=
Copyright (c) Benny Thörnberg 10:44
Laser triangulation
Copyright (c) Benny Thörnberg 11:44
Laser triangulation
Laser
H
r Object
Range reference level β
FZD
-D
D d
rMAX
Focal point of lens
Principle distance of
lens
C
Using two imaging
elements and one
orthogonally oriented laser
will reduce the risk of
optical occlusion.
+⋅+
−⋅−−⋅=
))tan(())tan((
))tan(())tan((1
ββ
ββ
dFDF
dFDFHr
ZDZD
ZDZD
Copyright (c) Benny Thörnberg 12:44
Laser triangulation
Laser
H
r Object
Range reference level β
FZD
-D
D d
rMAX
Focal point of lens
Principle
distance of
lens
C
+⋅+
−⋅−−⋅=
))tan(())tan((
))tan(())tan((1
ββ
ββ
dFDF
dFDFHr
ZDZD
ZDZD
+⋅+
−⋅−−⋅=
))tan(())tan((
))tan(())tan((1
ββ
ββ
DFDF
DFDFHr
ZDZD
ZDZDMAX
))tan(())tan((
))(tan1( 2
ββ
β
DFDFFH
d
r
ZDZD
ZD−⋅+
+⋅⋅≤
∂
∂
The largest derivate of r with respect to
d will occur when d=-D,
))tan()()tan((
))(tan1( 2
ββ
β
DFDFkdHFr
ZDZD
ZD−+
+⋅⋅∂⋅≤∂
Copyright (c) Benny Thörnberg 13:44
Center Of Gravity - COG
sNoOfColumnc
crI
crIr
cdNoOfRows
r
NoOfRows
r ≤≤
⋅
=
∑
∑
=
= 1,
),(
),(
)(
1
1
Copyright (c) Benny Thörnberg 14:44
Sub-pixel precision analysis
0 0.005 0.01 0.015 0.02 0.02516
18
20
22
24
26
28
30
Exposure time [s]
SN
R [d
B]
SNR vs Exposure time
SNR for a laser source
Copyright (c) Benny Thörnberg 15:44
Sub-pixel precision analysis
Speckle noise (multiplicative!)
F-number=5.6 F-number=40
Copyright (c) Benny Thörnberg 16:44
Sub-pixel precision vs Noise
H.C. van Assen, M. Egmont-Petersen and J.H.C. Reiber, “Accurate Object Localization in Gray
Level Images Using the Center of Gravity Measure: Accuracy Versus Precision”, IEEE Transactions
on Image Processing, Vol. 11, No. 12, Dec. 2002
Copyright (c) Benny Thörnberg 17:44
Thresholding
Copyright (c) Benny Thörnberg 18:44
Sub-pixel precision analysis
Thresholding
Copyright (c) Benny Thörnberg 19:44
Systematic error from thresholding
0.05 0.1 0.15 0.2 0.25 0.30
0.005
0.01
0.015
0.02
0.025
0.03
0.035
0.04
0.045
Threshold
Sta
nd
ard
de
via
tio
n o
f su
b-p
ixe
l e
rro
r
Simulated errorEstimated error
B.F. Alexander and Ng Kim Chew, ”Elimination of systematic error in subpixel accuracy
centroid estimation”, Optical Engineering, Vol. 30, No. 9, Sept. 1991
Copyright (c) Benny Thörnberg 20:44
Sub-pixel precision analysis
How to select threshold?
0 0.2 0.4 0.6 0.8 10.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
Threshold [Rate of max signal intensity]
Std
.de
v. o
f C
OG
ou
tpu
t
Subpixel precision versus threshold
Copyright (c) Benny Thörnberg 21:44
Sub-pixel precision analysis
Conclusions
•Higher values for the intensity thresholding results in lower
standard deviation for computed positions where the uncertainty of COG is stemming mainly from optical noise (noisy image)
•COG computation can benefit from choosing lower values for intensity thresholding if the analysed image has very little of noise.
This is because systematic error can become dominant.
•Threshold level must be selected based on a trade of between
systematic error and image noise
Copyright (c) Benny Thörnberg 22:44
Example – Wood Chip Analyzer
Range Imager
Collector
Chip feeder
W
V
Principles of a Wood Chip Analyzer
Copyright (c) Benny Thörnberg 23:44
A real world wood chip analyzer
Example – Wood Chip Analyzer
Copyright (c) Benny Thörnberg 24:44
Stereo imaging
x2,X2
y2,Y2
z2,Z2
(X2,Y2,Z2)
(x2,y2)λ
λ
(x1,y1)
y1,Y1
x1,X1
z1,Z1
(X1,Y1,Z1)
B
BXX
Zx
X
Zx
X
+=
−=
−=
12
22
2
11
1
)(
)(
λλ
λλ
World coordinate
Acquiring a 3D representation of a 3D
scene is a one to one mapping and can be done using two 2D imaging
elements.
Copyright (c) Benny Thörnberg 25:44
Stereo imaging
ZZZ == 21
Reference: R.C. Gonzales and R.E. Woods, Digital Image Processing, Addison-Wesley
)(
)(
21
11
Zx
BX
Zx
X
−=+
−=
λλ
λλ
12 xx
BZ
−−=
λλ
Copyright (c) Benny Thörnberg 26:44
Stereo imaging
12 xx
BZ
−−=
λλ
This equation implies that two corresponding points x1 and x2 must be found in
two different images of the same 3D scene. In literature, this operation is referred to as correspondence matching. This matching is preferable done
with correlation techniques (compare with Template matching from previous
lecture, SAD, SSD, NCC).
Distance between corresponding interest points x2–x1 give us the Disparity
map. Interest point = spatially local point with high gradient intensity.
Compute interest points � scan for pairs of similar interest points � compute
distance between pairs
Reference: C. Georgoulas et.al, “Real-time disparity map computation module”, Microprocessors and Microsystems,
32 (2008), Elsevier
Copyright (c) Benny Thörnberg 27:44
Stereo imaging
Left side camera Right side camera
Disparity map
Reference: image data set at Stanford Vision Lab
http://vision.stanford.edu/~birch/p2p/
12 xx
BZ
−−=
λλ
Further
computation of Z will give depth map
Copyright (c) Benny Thörnberg 28:44
Stereo imaging
�� = �� · ��� = �� · ��� , � are the pixel spacing
�� , �� are normalized pixel indexes
� = � − ���� − ��
� = � − � · ���� = � − � · ���
�� = �, �, � is the metric
coordinate for a world point.�, � is the metric coordinate
of a pixel.
Point cloud data � is a matrix of world coordinates,� = ���� , ���� , … , ���� ,
where � is the number of measured world coordinates.
A depth map is a matrix of computed depths, D= ���,�
Copyright (c) Benny Thörnberg 29:44
Stereo imaging
Left camera Right camera
Disparity Map
Point cloud data
Example of point cloud data
Copyright (c) Benny Thörnberg 30:44
Stereo imagingDepth resolution
From earlier slide, we have the formula for
computing depth, � = � − �������
Let us define disparity as �� − �� = �� ⋅ " ,
such that � = � − ��#$⋅%
Let us now derivate Z with respect to s, &'&% = ��
#$%� , giving an expression for how
much Z will change due to a small change on s, (� = (" ��#$%�
Consider the following example,� = 8*+�� = 3.5/+� = 5++
Copyright (c) Benny Thörnberg 31:44
Stereo imagingDepth resolution
0 5 10 15 20 250
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Depth Z
Depth
resolu
tion d
Z
Depth resolution for different sub-pixel precisions
0.1 pix
0.5 pix
1.0 pix
Great improvements of depth
resolution can be observed for
sub-pixel resolutions of the
disparity computation
Copyright (c) Benny Thörnberg 32:44
Stereo imagingAvoid correspondence matching at pixel level
Think of a stereo camera pair used for surveillance of an air space close to a wind turbine.
The goal is to be able to position any birds coming close to turbine.
We can first segment both right and left image to identify the bird as an image object.
Using the pixel position of the bird in left and right allows for triangulation of only the
image object. Hence, more computing spent on segmentation while less spent on stereo
matching.
Copyright (c) Benny Thörnberg 33:44
LED
driver
Time Of Flight Imaging
I-to-V
Transmitted signal
Received signal
Time of Flight imaging can be divided into two classes based on the method
used for modulation of light:
• Pulsed-light
• Light Detection and Ranging - LIDAR range scanners
• Flash LIDAR cameras
• Continuous Wave Modulation
• Phase-shift between emitted and received signal is used to indirectly measure time of flight
Copyright (c) Benny Thörnberg 34:44
Time Of Flight Imaging
By Driving_Google_Self-
Driving_Car.jpg: Steve
Jurvetsonderivative work:
Mariordo - This file was
derived from: Driving
Google Self-Driving
Car.jpg:, CC BY 2.0,
LIDAR
A set of laser light rays is used for
spatial sampling of range data in the
vertical direction. Continues
rotations are used for scanning a 360 degree view.
https://youtu.be/nXlqv_k4P8Q https://youtu.be/kIrMaPTENB0
Copyright (c) Benny Thörnberg 35:44
LED
driver
Time Of Flight Imaging
I-to-V
0 10 20 30 40 50 60-1
-0.5
0
0.5
1
1.5
2
Time [ ns ]
Voltage [
V ]
Transmitted signal
Received signal
Transmitted signal
Received signal
0
12�+32"452673�8"92:*3 = �2 = *
2<
A phase shift 0 of up to 360 degrees can be
measured uniquely. Correspondingly, the traveled distance of light is max �.
Longer distances will result in ambiguous data
Continuous wave modulation
Copyright (c) Benny Thörnberg 36:44
Time Of Flight Imaging
= > = 22 *?" 0 @ A>
Phase and amplitude measurements
Copyright (c) Benny Thörnberg 37:44
Time Of Flight Imaging
= > = 22 *?" 0 @ A>
Instead of expressing a time shift > for the cross correlation between two harmonics having the same angular frequency A, we express an angular shift B = A>= B = 2
2 *?" 0 @ BNext step is to sample the correlation function = B at four points of B equally spaced
over one period,
= 0D = 22 *?" 0
= 90D = − 22 "8: 0
= 180D = − 22 *?" 0
= 270H = 22 "8: 0
Phase and amplitude measurements
Copyright (c) Benny Thörnberg 38:44
Time Of Flight Imaging
Let us have a look at the following expression,
= 270D − = 90D= 0D − = 180H =
22 "8: 0 @ 22 "8: 022 *?" 0 @ 22 *?" 0 = 2 · "8: 0
2 · *?" 0 = tan 0
Thus, the phase shift 0 of the reflected harmonic signal can be computed as,
0 = 92:�� = 270D − = 90D= 0D − = 180H
Phase and amplitude measurements
Copyright (c) Benny Thörnberg 39:44
Time Of Flight Imaging
Let us have a look at a second expression,
= 270D − = 90D � @ = 0D − = 180H � = 2�"8:� 0 @ 2�*?"� 0 = 2�
Thus, the amplitude 2 of the reflected harmonic signal can be computed as,
2 = = 270D − = 90D � @ = 0D − = 180H �L
Phase and amplitude measurements
Copyright (c) Benny Thörnberg 40:44
Time Of Flight Imaging
Let us have another look at the expression for cross correlation between reflected signal and demodulation signal,
= > = lim�⇢Q R 1 @ 2 · *?" A9 − 0� �S
�� �S· *?" A9 @ A> �9
5 9 d 9 @ >Sampling the cross correlation signal = > at four points α = A> = 0D , 90D , 180D , 270Hmeans that we should demodulate the reflected signal with a demodulation signal having four different phase shifts
5(9)*?" A9 @ B
Integrator
R L
LReset
C(B)
Phase and amplitude measurements
Copyright (c) Benny Thörnberg 41:44
LED
driver
Time Of Flight Imaging
I-to-V
Phase
shiftB = 0D , 90D , 180D , 270D
BP
filter
LP
filterC(B)
Lock-in amplifiers for all pixels
Pixel
Four images = B having four different phase shifts B at demodulation are used to
compute an Amplitude image and a Phase image.
Copyright (c) Benny Thörnberg 42:44
Time Of Flight ImagingRange image to Point cloud
x,X
y,Y
z,Z
�� = �, �, �
(x,y)λ
� = 5 · ��� @ �� @ ��L
� = � · ��
� = � · ��
5
�, �, � is the metric
coordinate for a world point.�, � is the metric coordinate
of a pixel.5 is the time-of-flight range
from pixel to world coordinate
Point cloud data � is a matrix of world coordinates,� = ���� , ���� , … , ���� ,
where � is the number of measured world coordinates.
A range image is a matrix of range values, W = 5��,�
�� = �� · ��� = �� · ��� , � are the pixel spacing
�� , �� are normalized pixel indexes
Copyright (c) Benny Thörnberg 43:44
Time Of Flight ImagingExample: ToF camera from Melexis
Copyright (c) Benny Thörnberg 44:44