![Page 1: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/1.jpg)
Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video
Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier 1,2 Ivan Laptev 1,2, Nicolas Mansard 4, Josef Sivic 1,2,3
1 Inria 2 DI ENS, PSL 3 CIIRC, CTU in Prague 4 LAAS-CNRS
CVPR 2019, Long Beach, California
![Page 2: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/2.jpg)
Motivation: learn how to interact with objects
“Atlas” by Boston Dynamics
![Page 3: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/3.jpg)
GoalsInput: • A monocular RGB video
Output:• Person & object 3D poses • 3D contact forces
![Page 4: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/4.jpg)
2D pose estimation
[Cao et al., CVPR 2017]
Recognition stageMethod: a two-stage approach
![Page 5: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/5.jpg)
2D pose estimation
Method: a two-stage approach
![Page 6: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/6.jpg)
[He et al., CVPR 2016]
2D human pose
Contact recognizer
Recognition stageMethod: a two-stage approach
![Page 7: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/7.jpg)
Contact recognizer
Method: a two-stage approach
![Page 8: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/8.jpg)
[He et al., ICCV 2017]
2D human pose Contact states
Object 2D endpoint
estimation
Recognition stageMethod: a two-stage approach
![Page 9: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/9.jpg)
Object 2D endpoint
estimation
Method: a two-stage approach
![Page 10: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/10.jpg)
Person-object trajectory
optimization
[Tassa et al., IROS 2012][Carpentier et al., TRO, 2018]
2D object endpoints
2D human pose Contact states
Estimation stageRecognition stageMethod: a two-stage approach
![Page 11: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/11.jpg)
2D object endpoints
2D human pose Contact states
Recognition stage Estimation stage
Estimated 3D motion and forces
[Tassa et al., IROS 2012][Carpentier et al., TRO, 2018]
Person-object trajectory
optimization
Method: a two-stage approach
![Page 12: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/12.jpg)
Estimation stage
Object 3D posesPerson 3D poses
xx
minimizex, u, c ∑
e∈{person, object}∫
T
0le (x, u, c) dt
![Page 13: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/13.jpg)
Person-objectperson-groundcontact forces
uu
Estimation stage
![Page 14: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/14.jpg)
c
Contact positions
c
Estimation stage
![Page 15: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/15.jpg)
Subject to 1. Contact motion model:
Estimation stage
![Page 16: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/16.jpg)
Subject to: 2. Contact force constraints:
- Prevent the feet from sliding
Estimation stage
![Page 17: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/17.jpg)
Subject to: 3. Lagrangian dynamics equation
[Carpentier, et al. Pinocchio. https://stack-of-tasks.github.io/pinocchio. 2015-2019]
M( ) + b( , ) = τ( , )··xx ·xx u c
Estimation stage
![Page 18: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/18.jpg)
Experiments
![Page 19: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/19.jpg)
A new dataset: Handtool videos
Barbell Hammer Scythe Spade
![Page 20: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/20.jpg)
![Page 21: Estimating 3D Motion and Forces of Person-Object ... · Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier](https://reader034.vdocuments.net/reader034/viewer/2022043010/5fa0f156edf5bc4e6a153cad/html5/thumbnails/21.jpg)
Zongmian Li 1,2, Jiri Sedlar 3, Justin Carpentier 1,2
Ivan Laptev 1,2, Nicolas Mansard 4, Josef Sivic 1,2,3
At the poster (#202):
1 Inria 2 DI ENS, PSL 3 CIIRC, CTU in Prague 4 LAAS-CNRS
CVPR 2019, Long Beach, California
•Additional details•Quantitative results•Discussion