course 11tracker/media/pdf/siggraph2001_slides_11.pdfprovides angle measurements provides angle...
TRANSCRIPT
Course 11Course 11
Tracking: Beyond 15 Minutes of ThoughtTracking: Beyond 15 Minutes of Thought
IntroductionsIntroductions
Danette Danette AllenAllenGreg WelchGreg WelchGary BishopGary Bishop
Why thinking about tracking isso funWhy thinking about tracking isso fun
ItIt’’s a simple problem to states a simple problem to stateIt has a little of everythingIt has a little of everything
• A little physics
• A little electronics
• A little math
• A little signal processing
• A little programming
Why don’t you just…Why don’t you just…
•• Mount TV cameras on the walls?Mount TV cameras on the walls?
•• Use GPS?Use GPS?
•• Use MEMS accelerometers?Use MEMS accelerometers?
•• Use carbon Use carbon nanotubesnanotubes??
Con
fide
nce
Time
The 15 minute effect...The 15 minute effect...
GoalsGoals
•• To get you to the 15 minute point andTo get you to the 15 minute point andbeyond...beyond...
•• To equip you to evaluate the variousTo equip you to evaluate the variousofferings and understand the strengths andofferings and understand the strengths andweaknesses of each.weaknesses of each.
Tracking TechnologiesTracking Technologies
(Danette Allen)
Many ways to slice!Many ways to slice!
ConfigurationConfiguration• Outside-in vs. Inside-out
Type of measurementType of measurement• Absolute vs. Relative
• Range vs. angle
Physical mediumPhysical medium• Five categories
Five (Six) TechnologiesFive (Six) Technologies
InertialInertialAcousticAcousticMagneticMagnetic
MechanicalMechanicalOpticalOptical
Radio (GPS) is the sixth...Radio (GPS) is the sixth...• typically used outdoors
• not addressed in this course
Inertial TrackingInertial Tracking
PassivePassiveNewtonNewton’’s 2nd Law of Motions 2nd Law of Motion•F = ma (linear)
• ττττ = Iαααα (rotational)
No physical limits on working volumeNo physical limits on working volumeAccelerometers and gyroscopesAccelerometers and gyroscopes
• Derivative measurements
AccelerometersAccelerometers• Measure force exerted on a mass since we cannot measure
acceleration directly.
• Proof-mass and damped spring
• Displacement proportional to acceleration
• Potentiometric and piezoelectrictransducers
Inertial TrackingInertial Tracking
m
3 2 1 0 1 2 3
Inertial TrackingInertial Tracking
• Gimbal deflection
GyroscopesGyroscopes• Inertia
rigidity in space
• Precessiona comparatively slow gyration of the rotation axis of aspinning body about another line intersecting it so asto describe a cone (Mirriam-Webster)
(Discovery, 2001)
MicrogyroscopeMicrogyroscope
(Systron Donner, 2001)
MEMSMEMSTuning forkTuning fork
Ring-laserRing-laserand Fiberand Fiber
• Doppler effect
• Beat frequency
Axis of rotation
( )
→→→→⊕→→→
↓
xs
xs
xx
sa
ˆ1ˆ1ˆ?
&&&&&
εUser
LaPlace Transforms = σ + jω
Inertial DriftInertial Drift
Error accumulation due to integrationError accumulation due to integration• Poor SNR at low frequencies
• Inverse square weighting of noise
• Gravity vector misalignment• 1º tilt error over 10 seconds ⇒ 9m position error
Periodic recalibrationPeriodic recalibration• hybrid systems typical
Time [s] to 0.1 [m] ErrorTime [s] to 0.1 [m] Error
Effect of Sensor Noise on Inertial-Only Performance (time to 0.1 meters error)
Rate gyro dynamic range (5rad/s divided by noise) bits
Acc
eler
omet
er d
ynam
ic r
ange
(3G
div
ided
by
nois
e) b
its
The GeometryThe Geometry• The intersection of 2 spheres is a circle.
• The intersection of 3 spheres is 2 points.
• One of the two points easily eliminated
Speed of SoundSpeed of Sound• Varies with temperature and pressure
• ~ 331[m/s] in air at 0º C
• 1 ft/ms ⇒ SLOW!!
UltrasonicUltrasonic• 40 [kHz] typical
Acoustic TrackingAcoustic Tracking
Acoustic Tracking MethodsAcoustic Tracking Methods
Time of FlightTime of Flight• Measures the time required for a sonic pulse or pattern to
travel from a transmitter to a receiver.
• d [m] = v [m/s] * t [s], v = speed of sound (c)
• Absolute range measurement
Phase CoherencePhase Coherence• Measures phase difference between transmitted and received
sound waves
• Relative to previous measurement
• still absolute!!
Phase CoherencePhase Coherence
EquationsEquations• A cos(ωt - φ)
• c[m/s] = λ[m] * f[1/s]
• δ[m] = λ[m] * (φ/2π)
““RelativeRelative”” Result Result• Fractional wavelength
• Need previous range estimate
•No integration!!!
Magnetic TrackingMagnetic Tracking
Three mutually-orthogonal coilsThree mutually-orthogonal coils•
• Each transmitter coil activated serially
• Three measurements apiece(three receiver coils)
• Nine-element measurement for6D position
AC AC vsvs. DC. DC• Ferromagnetic interference
θπθθ
πsin
32cos
32 d
MH
d
Mr
H ==
Mechanical TrackingMechanical Tracking
Ground-based or body-basedGround-based or body-basedUsed primarily for motion captureUsed primarily for motion captureProvide angle and range measurementsProvide angle and range measurements
• gears
• bend sensors
Elegant addition of force feedbackElegant addition of force feedback
Optical TrackingOptical Tracking
Provides angle measurementsProvides angle measurements• One 2D point
defines a ray
• Two 2D pointsdefine a pointfor 3D position
• Additional 2Dpoints requiredfor orientation
Speed of lightSpeed of light• 2.998 * 108 m/s (1 ft/ns)
Cθ, φ
C
Active vs. Passive TargetsActive vs. Passive Targets
Typical detectorsTypical detectors• Video and CCD cameras
• Computer vision techniques
Passive targetsPassive targets• Reflective materials, high contrast patterns
CCD cell/pixel
Active vs. Passive TargetsActive vs. Passive Targets
Typical detectorsTypical detectors• LEPDs
•
• Quad Cells
•
Active targetsActive targets• LEDs
4 1
3 2
( ) ( ) ( ) ( )4321
3241
4321
4321
iiii
iiiiy
iiii
iiiix
++++−+=
++++−+=
( )( )
−=
−=
LxL
IIorL
xLII
0)sinh)(sinh
0 αα x
I
L
I ≈
Many ways to slice!Many ways to slice!
ConfigurationConfiguration• Outside-in vs. Inside-out
Type of measurementType of measurement• Absolute vs. Relative
• Range vs. angle
Physical mediumPhysical medium• Five categories
next
Source/Sensor ConfigurationsSource/Sensor Configurations
(Gary Bishop)(Gary Bishop)
Sensor ConfigurationsSensor Configurations
Geometric arrangement ofGeometric arrangement ofsensors and sources impacts:sensors and sources impacts:• accuracy
• usability
• algorithms
for example CODA mpx30for example CODA mpx30
3 1-D 3 1-D CCDs CCDs are stationaryare stationaryLED targets moveLED targets move
Very interesting optics and sensingVery interesting optics and sensing
CODA mpx30CODA mpx30
•• Measures angles in lab coordinate frameMeasures angles in lab coordinate frame
•• Angle determines a planeAngle determines a plane
•• Intersecting 3 planes determines a pointIntersecting 3 planes determines a point
“Flatland”
CODA mpx30CODA mpx30
Such Such ““outside-looking-inoutside-looking-in”” systems systems
•• measure position very wellmeasure position very well
•• allow many small moving targetsallow many small moving targets
•• use multiple targets to get orientationuse multiple targets to get orientation
•• trade off accuracy and working volumetrade off accuracy and working volume
•• provide larger volume / more accuracy withprovide larger volume / more accuracy withmore sensorsmore sensors
•• use really simple mathuse really simple math
HiBallHiBall
6 2-D sensors and 6 lenses in dodecahedron6 2-D sensors and 6 lenses in dodecahedron10001000’’s of s of LEDs LEDs fixed on the ceilingfixed on the ceilingCalibration gives effectively 26 camerasCalibration gives effectively 26 cameras
•• Measures angles in user coordinate frameMeasures angles in user coordinate frame
•• Angles determine a constraint relatingAngles determine a constraint relating• position
• orientation
• view
• led location
HiBallHiBall
“Flatland”
HiBallHiBall
Such Such ““inside-looking outinside-looking out”” systems systems•• directly measure orientationdirectly measure orientation•• allow large working volume with accuracyallow large working volume with accuracy•• are larger than LED targetsare larger than LED targets•• and thus harder to use for hands, feet, etc.and thus harder to use for hands, feet, etc.
Arc Second VulcanArc Second Vulcan
Sources scan Sources scan ““planes ofplanes oflightlight”” through space through space
Sensors on target detectSensors on target detectpassing planepassing plane
Sources
Sensors
Arc Second VulcanArc Second Vulcan
•• Time of plane passing converts to angle atTime of plane passing converts to angle atthe sourcethe source
•• Measures angles in world frameMeasures angles in world frame
•• Thus like CODA and other Thus like CODA and other ““outside-outside-looking-inlooking-in”” systems systems
•• Direction of Direction of ““lookinglooking”” really isn really isn’’t the issuet the issuebut coordinate frame of measurementbut coordinate frame of measurement
User and SensorUncertainty/Information
User and SensorUncertainty/Information
(Greg Welch)(Greg Welch)
Pose UncertaintyPose Uncertainty
•• Measurement uncertaintyMeasurement uncertainty• Pose estimates from noisy sensor measurements
•• User pose uncertaintyUser pose uncertainty• Noisy and temporally-discrete measurements
• Modeling user motion is difficult [Weber]
• Modeling pose uncertainty is less difficult
Noise-Driven ProcessesNoise-Driven Processes
∫Randomnoise ∫
Positionuncertainty
Velocityuncertainty
Randomnoise
Noisymeasurement
“True” sensormeasurement
Σ
User
Tracker
Random Variables and SignalsRandom Variables and Signals
•• Map sample space Map sample space →→→→→→→→ real numbers real numbers• For example, time to voltage
•• Random SignalsRandom Signals• For example, electrical signals
• Continuous random variables
• Probability over a region of sample space
• Spatial (statistical) and temporal (spectral) aspects
Cumulative Distribution FunctionCumulative Distribution Function
F x P xX ( ) = −∞( ],
1. as
2. as
3. is a non - decreasing function of
F x x
F x x
F x x
X
X
X
( ) → → −∞
( ) → →+∞
( )
0
1
Probability Density FunctionProbability Density Function
f xddx
F xx X( ) = ( )
1. is a non - negative function
2.
f x
f x dx
X
X
( )
( ) =−∞
∞
∫ 1
Probability (Continuous)Probability (Continuous)
P a b f x dxx xa
b
,[ ] = ( )∫
Statistical MomentsStatistical Moments
µmmE X= [ ] =
x f x dxmX ( )
−∞
∞
∫ x p xmX
x
( )∑Continuous: Discrete:
1st Moment or Mean1st Moment or Mean
µ = [ ] =E X
xf x dxX ( )−∞
∞
∫ xp xXx
( )∑Continuous: Discrete:
Central MomentsCentral Moments
c E Xmm= −( )[ ] =µ
x f x dxmX−( ) ( )
−∞
∞
∫ µ
Continuous: Discrete:
x p xmX
x
−( ) ( )∑ µ
2nd Central Moment or Variance2nd Central Moment or Variance
V X E X
E X
[ ] = −( )[ ]= [ ] −
µ
µ
2
2 2
“Mean of square minus square of mean”
Standard DeviationStandard Deviation
σ = [ ]V X
Gaussian/Normal DistributionGaussian/Normal Distribution
x →+∞−∞ µ
f xX ( )σ
f x eX
x
( ) =− −( )1
2
1
2 22
πσσ
µX ~ ,N µ σ 2( )
Autocorrelation (Time Domain)Autocorrelation (Time Domain)
RX τ()
0 ττ–
X1
X2
R E X t X tX τ τ( ) = ( ) +( )[ ]
Spectral Density (Frequency Domain)Spectral Density (Frequency Domain)
S j F R
R e d
X X
Xj
ω τ
τ τωτ
( ) = ( )[ ]
= ( ) −
−∞
∞
∫
The Wiener-Khinchine relationship
White Noise ProcessWhite Noise Process
RX τ()
0 ττ– 0 ω
SX jω()
RA
X ττ
( ) ==
if then
else
0
0S j AX ω( ) =
Growth in Pose UncertaintyGrowth in Pose Uncertainty
V X wdt[ ] = ∫0
w q~ ,N 0( )where
and “white.”
Control of Pose UncertaintyControl of Pose Uncertainty
Measurements ⇒ pose information
Sensor MeasurementsSensor Measurements
BreakBreak
(15 Minutes)(15 Minutes)
Traditional ApproachesTraditional Approaches
(Gary Bishop)(Gary Bishop)
Traditional Solution MethodsTraditional Solution Methods
•• Simple problem: Determine pose givenSimple problem: Determine pose givensensor readings.sensor readings.•• Linear algebra taught us aboutLinear algebra taught us about
N equations in N unknownsN equations in N unknowns•• Each equation is a Each equation is a constraintconstraint•• 3 DOF3 DOF��������3 constraints & 6 DOF3 constraints & 6 DOF��������66
constraintsconstraints••Unfortunately often non-linear constraintsUnfortunately often non-linear constraints
often with multiple solutionsoften with multiple solutions
Range TrackerRange Tracker
Intersect 3 spheres
Unfortunately, the solution is ugly...
SimplifySimplify
a. put mike 0 at origin
b. put mike 1 out X axis 1 unit
c. put mike 2 out Y axis 1 unit
d. all 3 mikes in Z=0 plane
Usual coordinate transform to convert to“lab” coordinates
Simpler Range EquationsSimpler Range Equations
Note ambiguity
Optical with fixed 1D sensorsOptical with fixed 1D sensors
For example, 1D CCD with razor blade castingFor example, 1D CCD with razor blade castinga shadowa shadow
••Calibrate to determine 3D plane equationCalibrate to determine 3D plane equationfrom sensor reading (non-trivial)from sensor reading (non-trivial)
•• For each sensor reading, write a linearFor each sensor reading, write a linearequation relating unknown x,y,z to planeequation relating unknown x,y,z to plane
•• Solve the system of equations for x,y,zSolve the system of equations for x,y,z
Solve...Solve...
Optical with fixed 2D sensorsOptical with fixed 2D sensors
For example, two video cameras looking atFor example, two video cameras looking atLEDs LEDs on the user.on the user.
•• Could treat as four 1-D sensorsCould treat as four 1-D sensors
OROR
•• Calibrate to get ray equation from u,vCalibrate to get ray equation from u,v
•• Rays wonRays won’’t intersect!t intersect!
•• Minimize distance between themMinimize distance between them
Set up equationsSet up equations
Ray equations
Distance
Minimum distance line must beperpendicular to both rays, so...
SolveSolve
Distance out eachray to closest point
Halfway between
B is the baseline
Stochastic ApproachesStochastic Approaches
(Greg Welch)(Greg Welch)
MotivationMotivation
•• Take into accountTake into account• Stochastic nature of sensor signals
• Varying amounts of sensor information
• Model of user motion
•• Combine sensor/measurement informationCombine sensor/measurement information• Combat (otherwise growing) pose uncertainty
• Fuse information from heterogeneous sensors
State-Space ModelsState-Space Models
Begin with difference equation for process
y a y a y uk k k n k k n k+ − − += + + +1 0 1 1, ,K
Re-write as
x
y
y
y
y
a a a a y
y
y
y
k
k
k
k
k n
n n k
k
k
k n
+
+
−
− +
− −
−
−
− +
≡
=
1
1
1
2
0 1 2 1
1
2
1
1 0 0 0
0 1 0 0
0 0
0 0 0 1 0
M
L
L
M O M
+
1
0
0
1
M
u
State-Space ModelsState-Space Models
x Ax Gu
y Hxk k k
k k
+ = +=
1
A xi G
x
y
y
y
y
a a a a y
y
y
y
k
k
k
k
k n
n n k
k
k
k n
+
+
−
− +
− −
−
−
− +
≡
=
1
1
1
2
0 1 2 1
1
2
1
1 0 0 0
0 1 0 0
0 0
0 0 0 1 0
M
L
L
M O M
+
1
0
0
1
M
u
Observer Design ProblemObserver Design Problem
x Ax Gu
z Hx vk k k
k k k
= += +
− −1 1
Process noiseMeasurement noise
Optimal EstimationOptimal Estimation
c a t b t t dtT
= ( ) ( )( )∫ cost , ,0
cost = −a b cost = −( )a b 2
Integral of AbsoluteValue of Error (IAE)
Integral ofSquare of Error (ISE)
The Kalman FilterThe Kalman Filter
R.E. Kalman, 1960R.E. Kalman, 1960• Recursive optimal estimator
•Minimum variance of error
• Versatile & robust
•Estimation•Sensor fusion
• Robotics, navigation, computer vision, economics, …
• http://www.cs.unc.edu/~welch/kalman/
• Java-Based Learning Tool, books, papers, etc.
• ACM SIGGRAPH 2001 tutorial (earlier today)
predictpredict correctcorrect
PREDICTPREDICT
x Axk k−
−= 1
P AP A Qk kT−
−= +1
transitionuncertainty
CORRECTCORRECT
x x K z Hxk k k k= + −( )− −
K P H HP H RkT
kT= +( )− − −1
P I KH Pk k= −( ) −
“denominator”(measurement space)
predictedactual
Hybrid Systems andMulti-Sensor FusionHybrid Systems andMulti-Sensor Fusion
magnetic coil
magnetic coil
magnetic coil
accelerometer
accelerometer
accelerometer
gyroscope gyroscope
gyroscope
Incremental EstimationA Single Constraint at a TimeIncremental EstimationA Single Constraint at a Time
C = constraints needed for a unique solutionM = constraints used per estimate updateC = constraints needed for a unique solutionM = constraints used per estimate update
M = CM = C
M > CM > C
1 < M < C1 < M < C
M = 1111M = 1111
Unique SolutionUnique Solution
Under-constrainedUnder-constrained
SCAATSCAAT
Over-constrainedOver-constrained
Inter-Estimate SummaryInter-Estimate Summary
MultipleConstraints
SCAAT
Measurements(constraints)
Time
Benefits of SCAAT ApproachBenefits of SCAAT Approach
Purposefully using minimal constraintsPurposefully using minimal constraints• Avoid simultaneity assumption
• Temporal improvements
• Simplicity and flexibility
• Online source/sensor autocalibration
• Can be applied to virtually any tracking system
•Measurement model for each type of sensor•Dynamic model for user motion (possibly trivial)
Bishop, 1982
Error SourcesError Sources
(Greg Welch)(Greg Welch)
Error in Head PoseError in Head Pose
•• Hard to fool Hard to fool ““mother naturemother nature””• Lifetime of visual experience and expectations
• Visual-proprioceptive conflicts
• Virtual-real misregistration
•• What to do?What to do?• Some amount of error is unavoidable
• Understand sources and seek to minimize
Error ClassificationError Classification
•• Pose estimate life cyclePose estimate life cycle• Noisy sensor measurement → Estimate →
Transport → Transform → Display
•• Two primary classes of errorTwo primary classes of error• Static (spatial)
• Delay-induced (temporal)
Static Measurement ErrorStatic Measurement Error
•• Static field distortionStatic field distortion• Repeatable error in the measurement data
• “Bias” that might be corrected via calibration
•• Random noise or jitterRandom noise or jitter• Non-repeatable error
• Random (electrical) noise such as described earlier
• Often dependent on the current pose
Pose-Dependent Noise (Example)Pose-Dependent Noise (Example)
Baseline noise ξ0and coefficientsa, b, and c weredetermined off line.
Delay-Induced ErrorDelay-Induced Error
•• Measurement validityMeasurement validity• Good at sample time, then old (aging)
• Finite, non-zero sample time
• Old sample ⇒ misregistration
•• Motion Motion predicitionpredicition• Measure where you are, but want where you will be
• Later w/ Bishop
The Simultaneity AssumptionThe Simultaneity Assumption
MeasureCompute
UserMotion
Time
Moderate arm & wrist translation
Moderate head rotation
1/2 [s] • 3 [m/s] • 20-80 [ms] ⇒ 1-10 [cm]
1/2 [s] • 180 [°/s] • 20-80 [ms] ⇒ 6-25 [cm](at arm’s length)
First-Order Dynamic ErrorFirst-Order Dynamic Error
ε θθdyn,˙= ∆t
εdyn, ˙x x t= ∆
Tracker + graphicspipeline latencyInstantaneous
velocities
Synchronization DelaySynchronization Delay
A.k.a. phase delay or rendezvous delay
PoseEstimateTimeline
Total Tracker LatencyTotal Tracker Latency
Sample the sensor(s)Estimate cyclesynchronization
Estimate the poseServer buffersynchronization
Network transportClient buffersynchronization
Total Tracker ErrorTotal Tracker Error
Static errorSimultaneity assumptionTotal tracker latencyRemaining latency
Closing (Error Sources)Closing (Error Sources)
•• Did I mention error magnification?Did I mention error magnification?
•• Consider the technologyConsider the technology• Understand its limitations
• Stay within the envelope
•• Prediction (next)Prediction (next)
Motion PredictionMotion Prediction
(Gary Bishop)(Gary Bishop)
Motion PredictionMotion Prediction
End-to-end delayEnd-to-end delay•• hurts in VR / hurts worse in ARhurts in VR / hurts worse in AR•• sourcessources
• time to measure pose
• delay in communicating pose
• application response to change
• graphics update
• display refresh
What to do about delay?What to do about delay?
1.1. MonitorMonitor
2.2. MinimizeMinimize
3.3. MitigateMitigate
Latency is not only a tracker problem.
But mitigation is best handled at the tracker.
Can prediction help?Can prediction help?
BlueBlue��������no predictionno predictionRedRed��������w/out inertialw/out inertialGreenGreen��������w/ inertialw/ inertial
Can prediction help?Can prediction help?
Limits to predictionLimits to prediction
Prediction error grows Prediction error grows quadraticallyquadratically with withmotion bandwidth and prediction intervalmotion bandwidth and prediction interval
Prediction ideasPrediction ideas
•• Extrapolate past behavior to the futureExtrapolate past behavior to the future
•• The more history the betterThe more history the better
•• Correlations in the Correlations in the usersusers coordinate frame coordinate frame
•• Inertial sensors helpInertial sensors help
•• Monitor |predicted Monitor |predicted –– actual| for tuning actual| for tuning
•• Use image shifting to reduce jitterUse image shifting to reduce jitter
ConclusionsConclusions
(Gary Bishop)(Gary Bishop)
Final ThoughtsFinal Thoughts
•• No silver bulletNo silver bullet• Tracking anywhere for any purpose is a dream
•• No free lunch, only tradeoffsNo free lunch, only tradeoffs• Energy / Accuracy / Bandwidth / Latency / Noise
•• No end in sightNo end in sight• Lots of possibilities for interesting work
• ReActor not based on any of the principles described here
ResourcesResources
•• http://www.http://www.cscs..uncunc..eduedu/~/~welchwelch//kalmankalman•• check out Course 8 notescheck out Course 8 notes•• Dozens of books on KF, here are a fewDozens of books on KF, here are a few
• “Optimal Estimation with an ...” by Lewis
• “Introduction to Random Signals...” by Brown
• “Kalman Filtering Theory and Practice” by Grewal
•• Beginnings of a tracking bibliographyBeginnings of a tracking bibliography
Exhibits we’re going to check outExhibits we’re going to check out
• 3rd Tech (cool demo)
• 5DT
• Ascension (ReActor is a new method)
• InterSense
• Measurand
• MetaMotion / PhoeniX Technologies / Vicon
• Polhemus
EndEnd