projecting computer graphics on moving surfaces: a simple calibration and tracking method , sketch...
DESCRIPTION
Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)TRANSCRIPT
![Page 1: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/1.jpg)
Projection Computer Graphics
on Moving Surfaces: A Simple Calibration and Tracking Method
Claudio Pinhanez MIT Media Laboratory
Frank Nielsen, Kim Binsted Sony Computer Science Laboratory
![Page 2: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/2.jpg)
Motivations
transforming every surface in
a space into a display
movable displays
enhancing performance (theater, story-telling, dance)
![Page 3: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/3.jpg)
Structure of the Problem
range
![Page 4: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/4.jpg)
Goal: Correction of the Projection
when the surface moves, the projection is corrected
![Page 5: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/5.jpg)
Problems
1. Calibrate the projector/camera relation
2. Track position/attitude of the surface
3. Correct in real time the projection
4. Compensate for oblique projection
[Raskar, Sketches’99]
![Page 6: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/6.jpg)
Problem 2.a:
Tracking the Position of the Surface
hard for vision-based
systems
(darkness)
“hacked” solution:
4 Infra-red LEDs
5th disambiguating LED
5th marker
![Page 7: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/7.jpg)
Problem 1: Calibration
From stereo vision:
the relationship between points on a planar
surface seen from 2 different cameras is a
homography (3x3 linear matrix) [Faugeras 93]
Geometrically,
pinhole cameras = pinhole projectors
![Page 8: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/8.jpg)
Camera/Projector Homography
camera projector
homography
c
c
c
p
p
p
z
y
x
HHc
z
y
x
p
cccc zyzxc , pppp zyzxp ,
![Page 9: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/9.jpg)
Calibration Process
Only 4 points!
Given:
Read:
Since P=HC,
4,3,2,11,, iyxp i
p
i
pi
4,3,2,11,, iyxc i
c
i
ci
1 TT CCPCH
TTTT ppppP 4321 TTTT ccccC 4321
(manual alignment)
![Page 10: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/10.jpg)
Problem 3:
Correcting the Projection
p=Hc
![Page 11: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/11.jpg)
![Page 12: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/12.jpg)
Delay and Noise
delay:
position at time t
projected at time t+dt (100-200ms)
sensor noise
Solution: Kalman filter
![Page 13: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/13.jpg)
![Page 14: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/14.jpg)
Problems
1. Calibrate the projector/camera relation
2. Track position/attitude of the surface
3. Correct in real time the projection
4. Compensate for oblique projection
[Raskar, Sketches’99]
![Page 15: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/15.jpg)
Problem 2.b: Estimating the Attitude
4 co-planar points -> unique solution
(4-perspective inversion)
high degree polynomial, very sensitive to physical
parameters -> unstable solution
attitude computed from ratio of
line segments + linear regression
in practice, better and faster
than iterative solution 1/1=1 0.7/1.4=0.5
![Page 16: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/16.jpg)
HyperMask [Millenium Motel]
white facial mask
with IR markers camera
projector
mask
![Page 17: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/17.jpg)
Attitude/Position Tracking
![Page 18: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/18.jpg)
Multiple Characters
![Page 19: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/19.jpg)
Facial Expressions
![Page 20: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/20.jpg)
Siggraph’ 99 Video
![Page 21: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/21.jpg)
(Lip Synching)
sound-based predictive lip synching
neural network: predicts the position of
mouth from the sound stream
Shigeo Morishima (Seikei University)
![Page 22: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/22.jpg)
HyperMask: Performances
Tomorrow, 9 am to 1 pm
Emerging Technologies
Millenium Motel
![Page 23: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/23.jpg)
Questions and Answers(?)
More info:
http://www.csl.sony.co.jp/
person/nielsen/HYPERMASK/hypermask.html
?
?
? ?
? ?
![Page 24: Projecting Computer Graphics on Moving Surfaces: A Simple Calibration and Tracking Method , Sketch & Applications, Novel Projection Methods track (SIGGRAPH 1999)](https://reader034.vdocuments.net/reader034/viewer/2022042813/5482f2e5b4af9f690d8b48f0/html5/thumbnails/24.jpg)