effects of surface characteristics on alignment of real and graphic objects in stereoscopic...
TRANSCRIPT
Effects of Surface Characteristics on Alignment of Real and Graphic
Objects in Stereoscopic Augmented Reality Environments
Ming Hou
Ergonomics in Teleoperation and Control (ETC) Laboratory
Department of Mechanical and Industrial Engineering
University of Toronto
January 6th, 2003
3D Measurement in Unmodelled World
Virtual Tape Measure
Problem:
Unknown relationship between real and virtual objects in Augmented Reality displays
Research on Real World Targets
Video ImageVirtual Pointer
LINE
AREA
VOLUME
Hemisphere
Cylinder
Pseudo-Transparency Phenomenon
Real ObjectReal Object
(in Video)
Virtual pointer inVirtual pointer in frontfront of surfaceof surface
VirtualVirtual pointer pointer behind behind realreal surfacesurface
““Transparency EffectTransparency Effect””Breakdown of Fusion Breakdown of Fusion + Fusion Conflict + Fusion Conflict
Fuse Real Surface Fuse Virtual Pointer
Virtual pointer inVirtual pointer in frontfront of surfaceof surface
Virtual Virtual pointer pointer behindbehind real surface real surface
Conflict between Binocular Disparity and Occlusion Cues
Theory of Surface Interaction: Fusion Breakdown depends on Texture Density
Low Texture Density High Texture Density
Virtual pointer inVirtual pointer in frontfront of surfaceof surface
Virtual Virtual pointerpointer behindbehind real surface real surface
Research Motivation
Is this conflict really significant?
If yes, can it be used as an extra cue for detecting interactions between real and virtual objects,
and thus locating the real objects in AR environment more easily and accurately?
Main Hypotheses #1Main Hypotheses #1
More accurate to indicate position More accurate to indicate position on (curved) surface with on (curved) surface with high high texture density (HTD) (HTD) than with than with low low texture density (LTD) (LTD)
HTDHTD better than better than LTDLTD
Main Hypotheses #2Main Hypotheses #2
OrientationOrientation of observer relative to of observer relative to target surfacetarget surface will have influence will have influence
CentreCentre different fromdifferent from Off-centreOff-centre
Top ViewReal Surface
Virtual Pointer
Stereo Camera
Top ViewReal Surface
Virtual Pointer
Stereo Camera
Main Hypotheses #3Main Hypotheses #3
FormForm of Virtual Pointer ( of Virtual Pointer (VPVP) ) will have impact on alignment will have impact on alignment performanceperformance
LINE LINE AREAAREA VOLUMEVOLUME
Main Hypotheses #4Main Hypotheses #4
BBinocular disparityinocular disparity (i.e. crossed (i.e. crossed vs 0 vs uncrossed) will affect vs 0 vs uncrossed) will affect alignment performancealignment performance
No disparity (ND) > Crossed (C) or Uncrossed (UC)No disparity (ND) > Crossed (C) or Uncrossed (UC)
NDNDCC UCUC
Experimental Investigation ofAR “ Surface Effects ”
Stereo CameraStereo Camera
Indigo 2Indigo 2
CylinderCylinder
BarrierBarrier
VirtualVirtual PointerPointer
Stereoscopic AR DisplayStereoscopic AR Display
SpaceballSpaceball
Methodology
Expt. #Factorial Design
Independent Variables
Dependent Variables (Measurement)
#1 2x2x3x3
Texture Density Surface Orientation Binocular Disparity
VP Orientation
Placement ErrorPlacement Error between perceived between perceived target and its actual target and its actual
position on real surfaceposition on real surface
#2 2x2x3x3
Texture Density Surface Orientation Binocular Disparity
VP Form
Placement Error Placement Error ++ Confidence Rating + Confidence Rating +
Preference RatingPreference Rating
#3 5x5Texture Density
Surface Orientation
Placement ErrorPlacement Error ++ Angular ErrorAngular Error between between estimated normal and estimated normal and
real surface normalreal surface normal
6 Images
(15 paired comparisons)
Virtual Pointer FormLINE AREA VOLUME
Texture
Density
High
Low
Subjective Comparisons
Ease of Use
Transparency
Ease of Fusion
Methodology
Expt. #Factorial Design
Independent Variables
Dependent Variables (Measurement)
Expt. #1
2x2x3x3
Texture Density Surface Orientation Binocular Disparity
VP Orientation
Placement ErrorPlacement Error between perceived between perceived target and its actual target and its actual
position on real surfaceposition on real surface
Expt. #2
2x2x3x3
Texture Density Surface Orientation Binocular Disparity
VP Form
Placement Error Placement Error ++ Confidence Rating + Confidence Rating +
Preference RatingPreference Rating
Expt. #3
5x5Texture Density
Surface Orientation
Placement ErrorPlacement Error ++ Angular ErrorAngular Error between between estimated normal and estimated normal and
real surface normalreal surface normal
Placement Error vs Texture Density - 1
Main Experimental Results - 1
VP Placement Error vs Texture Density(Error Bar = 95% CI, F(1,11) = 11.14, P = 0.007)
-0.4
0
0.4
0.8
1.2
1.6
High Low
Texture Density
Mean
Pla
cem
en
t E
rro
r (c
m) Farther
Closer
\ \ Cylinder Surface \ \
Placement Error along Surface Normal vs
Texture Density(Error Bar = 95% CI, F(4,40) = 41.34, P < 0.001)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0 10 20 30 40
Texture Density (%)
Me
an
Pla
ce
me
nt
Erro
r (
cm
)
Placement Error vs Texture Density - 2
Main Experimental Results - 2
Placement Error vs Surface Orientation - 1
Main Experimental Results - 3
VP Placement Error vs Target Position (Error Bar = 95% CI, F(1,11) = 98.19, P < 0.001)
-0.4
0
0.4
0.8
1.2
1.6
Centre Right
Target Position (relative to observer)
Mean
Pla
cem
en
t E
rro
r (c
m)
Closer
Farther
Placement Error along Surface Normal vs
Target Position(Error Bar = 95% CI, F(4,40) = 11.90, P < 0.001)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
1 2 3 4 5
Target Position on the Sphere
Mean
Pla
cem
en
t E
rro
r (c
m)
Placement Error vs Surface Orientation - 2
Main Experimental Results - 4
1 2 3 4 5
Altitude Error vs Target Position on Hemisphere
(Error Bar = 95% CI, F(4, 40) = 36.72, P < 0.001)
0
5
10
15
20
1 2 3 4 5
Target Position
Mean
Alt
itu
de E
rro
r
(deg
ree)
Angular Error vs Surface Orientation
Main Experimental Results - 5
1 2 3 4 5
Subjective Comparison Resultsw.r.t. Ease of Use and Ease of Fusion
Volume VP better than other VPs, regardless of texture density
w.r.t. Transparency
Volume VP + Highly textured least transparent combination
Main Experimental Results - 6
Texture Density High better than Low
Surface Orientation
Centre different from Off-Centre
Summary of Results - 1
VP Form VOLUME > AREA > LINE (subjective comparisons)
Binocular Disparity
Summary of Results - 2
NDNDCC UCUC
Conclusions - 1Conclusions - 1
Perceptual conflict does exist when real and virtual objects interact in 3D AR environments (a model proposed)
Perceptual conflict can be used as extraextra depth cue to indicate interaction between real and virtual objects
Optimal density value for Random Dot texture pattern was found as Engineering Solution for accurate 3D measurement
R-V Interaction Process ModelR-V Interaction Process Model
behind
Localisation Achieved
Fusiondifficulty
No fusion difficulty No
Yes
(Transparency)
at
at
behind
(Breakdown/Conflict)
Perception (Stereo Matching)
AR Display
User
External WorldDisplay of virtual pointer (VP)
superimposed upon real object surface VP Controller
Cognition / Decision Making(Stereo Matching + Cue Conflict Resolution)
VP behind, or at, real
surface?
VP behind, or at, real
surface?
Stereo Matching :
VP in front ofreal surface?
FusedImage?
Ad
just
men
t
Conclusions - 1Conclusions - 1
Perceptual conflict does exist when real and virtual objects interact in 3D AR environments (a model proposed)
Perceptual conflict can be used as extraextra depth cue to indicate interaction between real and virtual objects
Optimal density value for Random Dot texture pattern found as Engineering Solution for accurate 3D measurement
Conclusions - 2Conclusions - 2Target position does affect alignment
task: centrally located targets benefit performance, but have disadvantage when along the line of sight
Volumetric stereo graphic cursor (more fusable features along three dimensions) is subjectively the most favoured VP
“Pseudo-transparency” contributes literature of depth perception cues (shape-from-texture and stereo)
Implications for AR Interface DesignImplications for AR Interface Design
Random Dot Stereogram enhances 3D alignment performance
Perceptual conflicts can be used as extraextra depth cue to detect real object position
3D VP better than other VPs
Perceptual errors always exist
LimitationsLimitations Implementation
Display Mode : stationary display without motion parallax and motion
perspective
Binocular Disparity : confounded with size cue and resolution
Scope
VP Design: line thickness of wire-frame VP
Texture Pattern: square Random Dot pattern
Future Work/Impact - 1Future Work/Impact - 1
Near Term Research
Projected lighting : more practical Motion parallax with Video-HMD Computational vision may alleviate
some error (being investigated)Simulated Projector
Stereo Cameras
Projected lighting with random dot
texture pattern
Future Work/Impact - 2 Long Term Interests
Integration of Computer Assisted Object Detection in AR Displays See-through HMD AR for Dismounted Soldiers in the Battlefield (e.g., Perceptual Conflicts, Navigational Aids, etc.)
Other Human Computer Interaction (HCI) Topics (e.g., Integration of Electronic Information
with Human-Machine Systems, etc.)
AcknowledgementAcknowledgement• Dr. Julius Grodski at Defence Research &
Development Canada (DRDC) – Toronto, Prof. Allison B. Sekuler and Prof. Paul Milgram at University of Toronto
• Dr. Stephen Ellis at NASA and Prof. Stanley Hamstra at University of Toronto
• Natural Sciences and Engineering Research Council (NSERC) Doctoral Scholarship
• Institute of Robotics and Intelligent Systems (IRIS), Canada
• Ontario Graduate Scholarship (OGS)
Experimental #1 Main ResultVP Placement Error vs Surface Texture
(Error Bar = 95% CI, F(1,9) = 619.71, P< 0.001)
-9
-8
-7
-6
-5
-4
-3
-2
-1
0
1
High Low
Texture Density
Pla
ce
me
nt
Err
or
(cm
)Farther
Closer
\ \ Cylinder Surface \ \
VP Form and Orientation in 1st Experiment :
Diagonal Horizontal
Vertical
Observer
Interaction in Experiment # 1Interaction in Experiment # 1
Interaction between Surface Texture (High vs Low) and Target Position (Center vs Right)
VP Placement vs Texture and Target Position
(Error Bar = +/- 1SD, F(1,9) = 246.33, P< 0.001)
-11-10-9-8-7-6-5-4-3-2-101
Centre Right
Angular Displacement of Target Normal
Pla
cem
ent E
rro
r (c
m)
High
Low
Paired Comparison Result
Ease of Use
Transparency
Ease of Fusion
Image # 1 2 3 4 5 6
Mean Z score
0 1.15 1.71 2.26 3.16
Image # 2 1 3 6 5 4
0 0.25 0.89 2.97 3.82 4.37
Image # 1 2 3 5 4 6
0 0.69 0.73 2.08 2.14 2.26
Measurement of Placement Error along Surface Normal
Estimated Target
Positive Error
Real Surface Target
Surface Normal
Positive error shows the estimated target is inside the sphere along surface normal
Placement Error vs Texture Density for Experiments 2 and 3
Placement Error along Surface Normal vs
Texture Density(Error Bar = 95% CI, F(4,40) = 41.34, P < 0.001)
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
0 10 20 30 40
Texture Density (%)
Me
an
Pla
ce
me
nt
Erro
r (
cm
)
Experiment 2Experiment 3
Definition of Angular Error
X
Y
P
N
T
Z
O
Q
RS
'
'
Altitude Error : Angular distance between estimated normal TP and real surface normal TN
Azimuth Error : Angular Error between horizontal projection OR and OS
'
'
Angular Bias in Spherical Coordinate
Y
Bias Area
S
E
N
X
Z
T
Angular bias tilted upwards from real surface normal (TN)
Example of Distribution of Altitude and Azimuth
Angular bias between estimated and real surface normal
Altitude Error vs Azimuth Error at Target 5
-30
-20
-10
0
10
20
30
40
50
60
-180 -135 -90 -45 0 45 90 135 180
Azimuth Error (degree)
Alt
itu
de
Err
or
(deg
ree)
Side View
Stereo Cameras
Ground Truth Measurements in Real SceneCylinder Stimulus
Real Distance in Depth (Z)
Cylinder Calibration objectIron plate
Top View
Real World Origin (0,0,0)
Registration Verification: Measurement of a Pin and a Cube
Calibration Target
Stereo Cameras
Calibration Cube
VP Resolution for Experiment 2 VP Resolution and Accuracy TestsVP Resolution (Percentage of Real Distance
vs One Pixel Error of VP in Display)
0.00%
1.00%
2.00%
3.00%
4.00%
5.00%
6.00%
7.00%
8.00%
0 1 2 3 4 5
Distance of Object from Cameras (m)
Perc
en
tag
e o
f D
ista
nce (
D)
for
1
Pix
el C
han
ge in
Ho
rizo
nta
l D
isp
ari
ty
Retinal Disparity in Stereoscopic Display for One Pixel Separation (exaggerated)
b
a
C’
d
ZD
XD
2e
P = 1 pixel separation
Apparent position of point C’, due to 1 pixel horizontal disparity
StereoscopicDisplay Monitor
Viewer’sEye
b /2
)2
)2(2)(2
d
Pearctg
d
earctgba
Psychophysical Standard for Texture Density Control
• Spatial scale (size)
• Homogeneity (spatial regularity, density is approximately constant over the surface)
• Isotropy (no orientation bias, equally to be oriented in all directions) – compression
Practical Augmented Reality Example
Distance between point 1 and point 2 is 7mm
Coordinate of Point 1 (2.3, 14.7, 96.2)
Coordinate of Point 2 (1.8, 14.4, 95.8)
Virtual Tape Measure for Minimally Invasive
Surgery