ilaria gori, jivko sinapov, priyanka khante, peter stone ...jsinapov/slides/icsr_2016_sldes.pdf ·...
TRANSCRIPT
![Page 1: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/1.jpg)
Robot-Centric Activity Recognition 'in the Wild'
Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone and J. K. Aggrawal
University of Texas at Austin, Austin TX 78712, USA{ilaria.gori,aggarwaljk}@utexas.edu,
{jsinapov,pkhante,pstone}@cs.utexas.edu
![Page 2: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/2.jpg)
3
![Page 3: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/3.jpg)
4
![Page 4: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/4.jpg)
5
Motivation
“taking a picture”
![Page 5: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/5.jpg)
6
Related Work
(Ryoo and Matthies 2013)(Xia et al. 2011)
(Ryoo et al. 2015)
![Page 6: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/6.jpg)
7
Limitations of Existing Work
● The activities were specified by the researchers ahead of the experiment
● The activities were performed by a small number (5 to 8) of 'actors'
● The robot is either stationary or teleoperated
![Page 7: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/7.jpg)
8
Dataset Collection
![Page 8: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/8.jpg)
9
Video
![Page 9: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/9.jpg)
10
Dataset Collection
● Robot: Segbot● Environment: 3rd Floor of GDC, spanning a public
undergraduate lab and a graduate lab● The robot autonomously traversed the environment
for 1-2 hours a day over the course of 6 days covering ~14 km total
● Whenever the robot's Kinect 2.0 detected a person, the robot recorded a range of visual and non-visual data which was later used for classification
![Page 10: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/10.jpg)
11
Example Human Detection
![Page 11: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/11.jpg)
12
Example Human Detection
. . . . . .
![Page 12: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/12.jpg)
13
Recorded Data
![Page 13: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/13.jpg)
14
Recorded Data
Dataset size: ~ 140 GBAvailable upon request
![Page 14: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/14.jpg)
15
Activity Labels
![Page 15: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/15.jpg)
16
System Overview
![Page 16: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/16.jpg)
17
Visual Features
● Histogram of 3D Joints (HOJ3D)● Covariance of Joint Positions over Time (COV)● Histogram of Direction Vectors (HODV)● Histogram of Oriented 4D Normals (HON4D)● Pairwise Relational Matrix (PRM)
![Page 17: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/17.jpg)
18
Additional Features
● Human-Robot Velocity Features: The direction in which the human moves with respect to the robot
● Distance Features: The distance between the human and robot over time
● Localization Features: The robot's pose (position and orientation) in the map
![Page 18: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/18.jpg)
19
Example Feature Sequence
xvis
(t) xvis
(t+1) . . .xvis
(t+2) xvis
(t+k)
xvel
(t) xvel
(t+1) . . .xvel
(t+2) xvel
(t+k)
xdis
(t) xdis
(t+1) . . .xdis
(t+2) xdis
(t+k)
xloc
(t) xloc
(t+1) . . .xloc
(t+2) xloc
(t+k)
Visual:
Velocity:
Distance:
Location:
![Page 19: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/19.jpg)
20
Feature Quantization
xvis
(t) xvis
(t+1) . . .xvis
(t+2) xvis
(t+k)
Quantization
![Page 20: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/20.jpg)
21
Feature Quantizations
● The computed features for each descriptor were quantized using k-means
● Bag-of-Words representation was obtained by counting the occurrence of each “word” over the course of each video
● The BoW representations of all descriptors were concatenated to obtain a final feature vector
![Page 21: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/21.jpg)
22
Evaluation
● Evaluation was performed using 5-fold cross validation
● Because the dataset was unbalanced, the kappa statistic was used to measure performance
Probability of correct classification by classifier
Probability of correct classification by chance
![Page 22: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/22.jpg)
23
Classification Results
Vision Only Vision + Distance + Velocity
COV [6] 0.329 0.440
HOJ3D [16] 0.515 0.633
HODV [3] 0.624 0.649
PRM 0.547 0.660
HON4D [11] 0.756 0.762
![Page 23: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/23.jpg)
24
Can the robot exploit the spatial structure of activities?
![Page 24: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/24.jpg)
25
“false detection”
“wave”
“sit”
“walk away”
Can the robot exploit the spatial structure of activities?
![Page 25: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/25.jpg)
26
Classification Results
Vision Only Vision + Distance +
Velocity
Vision + Distance + Velocity + Localization
COV [6] 0.329 0.440 0.462
HOJ3D [16] 0.515 0.633 0.651
HODV [3] 0.624 0.649 0.660
PRM 0.547 0.660 0.671
HON4D [11] 0.756 0.762 0.764
![Page 26: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/26.jpg)
27
Summary and Conclusion
● Conducted largest experiment in robot-centric activity recognition to-date
● Dataset is available upon request● Evaluated 5 different visual features ● Demonstrated that non-visual features can
improve classification results
![Page 27: Ilaria Gori, Jivko Sinapov, Priyanka Khante, Peter Stone ...jsinapov/slides/ICSR_2016_sldes.pdf · Robot-Centric Activity Recognition 'in the Wild' Ilaria Gori, Jivko Sinapov, Priyanka](https://reader034.vdocuments.net/reader034/viewer/2022050405/5f8310591057c775b87f50dc/html5/thumbnails/27.jpg)
28
Thank you!
Ilaria Gori Jivko Sinapov Priyanka Khante Peter Stone J.K. Aggarwal
http://www.cs.utexas.edu/~larg/bwi_web/