mixed scale motion recovery james davis ph.d. oral presentation advisor – pat hanrahan aug 2001
Post on 19-Dec-2015
217 views
TRANSCRIPT
Mixed Scale Motion Recovery
James DavisJames Davis
Ph.D. Oral PresentationPh.D. Oral Presentation
Advisor – Pat HanrahanAdvisor – Pat Hanrahan
Aug 2001Aug 2001
3
• Acquire real motion as computer modelAcquire real motion as computer model
• Fixed resolution vs. range ratioFixed resolution vs. range ratio
Current technology
4
Applications of motion recovery
• AnimationAnimation
• Athletic analysisAthletic analysis
• BiomechanicsBiomechanics
5
Mixed scale domains
• Detailed motion within a larger volumeDetailed motion within a larger volume
6
Problem domain characterization
• Multiple scales of motionMultiple scales of motion
• At individual scalesAt individual scales• Working volume is local
• Working volume is moving
7
Hierarchical paradigm
• Explicitly expresses motion hierarchyExplicitly expresses motion hierarchy• Motion recovery drives sub-region selection
• Sub-region selection defines next scale
• Multiple designs possible within frameworkMultiple designs possible within framework
Motion recovery
Sub-region selection
Motion recovery
Motion recovery
Sub-region selection
Large scale Medium scale Small scale
8
My design• Multi-camera large scale recoveryMulti-camera large scale recovery
• Covers large volume
• Robust to occlusion
• Pan-tilt multi-camera small scale recoveryPan-tilt multi-camera small scale recovery• Automated camera control
• High resolution imaging
9
Demonstration of my system
• Body moves on room size scaleBody moves on room size scale
• Face deforms on much smaller scaleFace deforms on much smaller scale
• Simultaneous captureSimultaneous capture
10
Desirable system properties
• High resolution/range ratioHigh resolution/range ratio
• Scalable Scalable
• Occlusion robustnessOcclusion robustness
• Runtime automationRuntime automation
11
Related Work
• Traditional motion recoveryTraditional motion recovery• [Vicon] [MotionAnalysis] [Guenter98]
• Simple pan/tilt systemsSimple pan/tilt systems• [Sony EVI-D30] [Fry00]
• 2D guided pan/tilt systems2D guided pan/tilt systems• [Darrell96] [Greiffenhagen00]
• Human controlled camerasHuman controlled cameras• [Kanade00]
Runtime automation
High resolution/range ratio
Scalable
Occlusion robustness
Recovers motion
12
Contributions
• Framework for mixed scale motion recoveryFramework for mixed scale motion recovery• Hierarchical paradigm
• Data driven analysis
• Model based solutions
• Specific system designSpecific system design• High resolution/range ratio
• Scalable
• Robust to occlusion
• Automated
• Application to simultaneous face-body captureApplication to simultaneous face-body capture
13
Talk outline
• IntroductionIntroduction
• Framework Framework • Hierarchical paradigm
• Data driven analysis
• Model based solutions
• System implementationSystem implementation• Large scale recovery
• Sub-region selection
• Small scale recovery
• End-to-end videoEnd-to-end video
• Summary and future workSummary and future work
14
Relation of system to hierarchy
Large scalemotion recovery
Small scalemotion recovery
Sub-region selection
15
System overview
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
Large scalemotion recovery
Small scalemotion recovery
Sub-region selection
16
Interface is the challenge
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
• Large/small scale similarLarge/small scale similar
• Interface requirements differInterface requirements differ
• Interface critical in end-to-end systemInterface critical in end-to-end system
• Often ignored in individual componentsOften ignored in individual components
Interfaces
17
Data flow
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
• System viewed as data flowSystem viewed as data flow
• Clean interface (data) desirable Clean interface (data) desirable
18
System challenges
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
Occlusion
Noise
Unknown correspondence
Incorrect camera model
Latency
Occlusion
Unknown camera motion
19
Model improves data
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
Occlusion
Noise
Unknown correspondence
Incorrect camera model
Latency
Occlusion
Unknown camera motion
Model
Model
Model
Model
Model
Model
20
Model improves data
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
Occlusion
Noise
Unknown correspondence
Incorrect camera model
Latency
Occlusion
Unknown camera motion
Kalman filter
P/T camera model
Prediction
Face model
21
Talk outline
• IntroductionIntroduction
• Framework Framework • Hierarchical paradigm
• Data driven analysis
• Model based solutions
• System implementationSystem implementation• Large scale recovery
• Sub-region selection
• Small scale recovery
• End-to-end videoEnd-to-end video
• Summary and future workSummary and future work
22
Large scale system
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
23
Large scale physical arrangement
• 18 18 NTSCNTSC cameras cameras
• 18 digitizing Indys18 digitizing Indys
25
Large scale pose recovery
• Consider rays through observationsConsider rays through observations
• Rays cross at 3D feature pointsRays cross at 3D feature points
26
Calibrating wide area cameras
• Jointly calibrated multiple camerasJointly calibrated multiple cameras
• Iteratively estimateIteratively estimate• Camera calibration
• Target path
[Chen, Davis 00]
27
Unknown correspondence
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
Unknown temporal correspondenceKalman filter
28
Unknown temporal correspondence
• Multiple 3D features recoveredMultiple 3D features recovered
• Which feature is the head?Which feature is the head?
• Each frame is independently derivedEach frame is independently derived
29
Dynamic motion model
• Not single frame triangulationNot single frame triangulation
• Dynamic motion modelDynamic motion model• Model continuous motion
• Update on each observation
• Estimate position/velocity
• Extended Kalman filter
[Kalman 60] [Broida86] [Welch97]
30
Benefits of motion model
• Feature Feature IDsIDs maintained maintained
• Robust to short occlusionRobust to short occlusion
• Synchronized cameras unnecessary Synchronized cameras unnecessary
31
Sub-region selection
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
32
Simplistic camera model
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
Incorrect camera model
P/T camera model
33
Simplistic camera model
• Pan/tilt axes not aligned with optical centerPan/tilt axes not aligned with optical center
IxIy
= C Ry Rx
xyz
34
New camera model
• Arbitrary pan/tilt axesArbitrary pan/tilt axes
• Jointly calibrate axes and cameraJointly calibrate axes and camera• Observe known points from several pan/tilt settings
• Fit data with minimum error
= C Tpan Rpan Tpan Ttilt Rtilt Ttilt
IxIy
xyz
-1-1
[Shih 97]
35
Latency
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
LatencyPrediction
36
Camera motor latency
• Empirically found 300ms latencyEmpirically found 300ms latency
• High velocity targets leave frameHigh velocity targets leave frame
• Prevents accurate sub-region selectionPrevents accurate sub-region selection
37
Target motion prediction
• Predict future target motionPredict future target motion
• Point camera at predicted target locationPoint camera at predicted target location
• Use previous motion modelUse previous motion model
• High velocity objects successfully trackedHigh velocity objects successfully tracked
P’ = Pi + t • Vi
38
Small scale system
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
39
Small scale physical arrangement
• 4 Pan-tilt cameras point at sub-region4 Pan-tilt cameras point at sub-region
• 4 SGI O2s digitize video4 SGI O2s digitize video
40
Small scale features
• Painted face featuresPainted face features
• Image gradient feature trackingImage gradient feature tracking
[Lucas,Kanade 81] [Tomasi, Kanade 91]
42
Problems with face recovery
Feature Tracking
3D Pose Recovery
Video streams
2D points
Pan/tilt parameters
Pan/tilt controller
LEDs
3D points
Feature Tracking
3D Pose Recovery
Video streams
2D points
3D points
Occlusion
Unknown camera motion
Face model
43
Problems with face recovery
• Self occlusionSelf occlusion• Many points not visible
• Camera motion not known preciselyCamera motion not known precisely• Difficult to merge more than two views
Recovered 3D geometryView from one camera
44
Face model
• Face model defines the set of valid facesFace model defines the set of valid faces• Linear combination of basis faces
• Capture basis set under ideal conditions
• Basis transformation
FF = = wwii BBii
= w1 w2 w3+ +
[Turk, Pentland 91] [Blanz,Vetter 99] [Guenter et.al. 98]
45
Model evaluation
Mean error < 1.5 mmMean error < 1.5 mm
Remove
features
Fit to
model
Reconstruct
features
Observe
everything
Evaluate
error
46
Reconstructed face
• Fit partial data to the modelFit partial data to the model• Use model to reconstruct complete geometryUse model to reconstruct complete geometry
Recovered
geometry
from video
Reconstructed
geometry
from model
48
Talk outline
• IntroductionIntroduction
• Framework Framework • Hierarchical paradigm
• Data driven analysis
• Model based solutions
• System implementationSystem implementation• Large scale recovery
• Sub-region selection
• Small scale recovery
• End-to-end videoEnd-to-end video
• Summary and future workSummary and future work
49
Summary of contributions
• Framework for mixed scale motion recoveryFramework for mixed scale motion recovery• Hierarchical paradigm
• Data driven analysis
• Model based solutions
• Specific system designSpecific system design• High resolution/range ratio
• Scalable
• Robust to occlusion
• Automated
• Application to simultaneous face-body captureApplication to simultaneous face-body capture
50
Future directions
• Application to other domainsApplication to other domains
• More levels of hierarchyMore levels of hierarchy
• Selection of multiple sub-regionsSelection of multiple sub-regions
• Alternate system designsAlternate system designs
51
Acknowledgements
Prof. Pat Hanrahan,Prof. Pat Hanrahan, Prof. Brian Wandell, Prof. Brian Wandell, Prof. Chris Bregler, Prof. Gene Alexander, Prof. Chris Bregler, Prof. Gene Alexander, Prof. Prof. Marc Levoy, Cindy Chen, AdaMarc Levoy, Cindy Chen, Ada Glucksman, Heather Gentner, Homan IgehyGlucksman, Heather Gentner, Homan Igehy, , Venkat Krishnamurthy,Venkat Krishnamurthy, Tamara Munzner, Tamara Munzner, FranFranççois Guimbretiois Guimbretièère,re, Szymon Rusinkiewicz,Szymon Rusinkiewicz, Maneesh Agrawala, Lucas Pereira, Maneesh Agrawala, Lucas Pereira, Kari Pulli, Kari Pulli, Shorty, Sean Anderson, Reid Gershbein, Philipp Shorty, Sean Anderson, Reid Gershbein, Philipp Slusallek, Milton Chen, Mathew Eldridge, Natasha Slusallek, Milton Chen, Mathew Eldridge, Natasha Gelfand, Gelfand, Olaf HallOlaf Hall--Holt, Humper, Brad Johanson, Sergey Brin, Holt, Humper, Brad Johanson, Sergey Brin, Dave Koller, John Owens, Dave Koller, John Owens, Kekoa ProudfootKekoa Proudfoot,, KathyKathy Pullen, Bill Mark, Dan Pullen, Bill Mark, Dan Russel, Larry Page, Li-Yi Wei, Russel, Larry Page, Li-Yi Wei, Gordon Stoll, Julien Gordon Stoll, Julien BaschBasch, , Andrew Beers, Hector Andrew Beers, Hector Garcia-Molina,Garcia-Molina, Brian Freyburger,Brian Freyburger, Mark Horowitz,Mark Horowitz, Erika ChuangErika Chuang, , Chase Garfinkle,Chase Garfinkle, John Gerth, John Gerth,
Xie Feng,Xie Feng, Craig Kolb,Craig Kolb, ToliToli, , Mom, Dad,Mom, Dad, Holly Jones, Chris, Crystal, Lara, Holly Jones, Chris, Crystal, Lara, Grace Gamoso,Grace Gamoso, Matt Hamre,Matt Hamre, Nancy Schaal,Nancy Schaal, Aaron Jones,Aaron Jones, Bandit, Xiaoyuan Bandit, Xiaoyuan TuTu, , Abigail, Shefali,Abigail, Shefali, Liza,Liza, Phil,Phil, Deborah,Deborah, Brianna, Alejandra, Miss Dungan, Gabe, Sedona, Sharon, Gonzalo, and many other children whose names I can no longer rememberBrianna, Alejandra, Miss Dungan, Gabe, Sedona, Sharon, Gonzalo, and many other children whose names I can no longer remember