![Page 1: CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil,](https://reader036.vdocuments.net/reader036/viewer/2022062409/5697bf7d1a28abf838c84b4b/html5/thumbnails/1.jpg)
CAMEO: Meeting Understanding
Prof. Manuela M. Veloso, Prof. Takeo Kanade
Dr. Paul E. Rybski, Dr. Fernando de la Torre,
Dr. Brett Browning, Raju Patil, Carlos Vallespi,
Scott Lenser, Betsy Ricker, Francesco Tamburrino,
Colin McMillen, Sonia Chernova
CALO: Physical Awareness
Computer Science Department /The Robotics Institute
School of Computer Science
Carnegie Mellon University
![Page 2: CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil,](https://reader036.vdocuments.net/reader036/viewer/2022062409/5697bf7d1a28abf838c84b4b/html5/thumbnails/2.jpg)
CAMEO: Camera Assisted Meeting Event Observer
![Page 3: CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil,](https://reader036.vdocuments.net/reader036/viewer/2022062409/5697bf7d1a28abf838c84b4b/html5/thumbnails/3.jpg)
CAMEO : Camera Assisted Meeting Event Observer
• Robust multi-person PA capture device
• Contributions• Mosaic generation• Person tracking• Face recognition• Activity recognition• Logging/modeling
Must effectively operate in unstructured environments.
Each camera is hand-calibrated only once to compensate for radial distortion
![Page 4: CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil,](https://reader036.vdocuments.net/reader036/viewer/2022062409/5697bf7d1a28abf838c84b4b/html5/thumbnails/4.jpg)
Video Mosaic
![Page 5: CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil,](https://reader036.vdocuments.net/reader036/viewer/2022062409/5697bf7d1a28abf838c84b4b/html5/thumbnails/5.jpg)
Person Tracking : Mean Shift Based Color Tracking
Register New Person: Person ID, Face histogramFace Center (x,y), Face width, Face height
• Additional filtering based on shape and color templates
“Omega” head and shoulder template
Henry Schneiderman. “Feature-Centric Evaluation for Cascaded Object Detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2004.
Henry Schneiderman. “Learning on Restricted Bayesian Network for Object Detection.” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2004.
![Page 6: CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil,](https://reader036.vdocuments.net/reader036/viewer/2022062409/5697bf7d1a28abf838c84b4b/html5/thumbnails/6.jpg)
Face Recognition: Training
1) Capture visual data2) Normalize for
geometry and illumination
3) Cluster the most discriminating face examples
Multiple face discrimination Real-time performance
![Page 7: CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil,](https://reader036.vdocuments.net/reader036/viewer/2022062409/5697bf7d1a28abf838c84b4b/html5/thumbnails/7.jpg)
classes
i
Trj
ri
classes
ij Cr Cr
ri
rj
rj
ri
ri
rj
ri
rj
T
i j
tr1
1111)))())(((( 21
1 2
21211212 BμμΣΣμμΣΣΣΣB
Face Recognition: Non-Linear Oriented Discriminant Analysis
Find transformation matrix B that MAXIMIZES the Kullback-Leibler divergence between clusters among classes
classes
ii
Ti
TtrJ1
1 ))()(()( BABBΣBBUse Iterative Majorization to approximate B
Project clustered images into a lower-dimensional subspace to speed recognition
Research challenge: Face subspace is multi-modal
![Page 8: CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil,](https://reader036.vdocuments.net/reader036/viewer/2022062409/5697bf7d1a28abf838c84b4b/html5/thumbnails/8.jpg)
Face Recognition: Results
• Each new face is projected into subspace and compared against the trained examples
• Closest match, via Mahalanobis distance, determines class membership
• 95% recognition rate with training database of 11 subjects
![Page 9: CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil,](https://reader036.vdocuments.net/reader036/viewer/2022062409/5697bf7d1a28abf838c84b4b/html5/thumbnails/9.jpg)
Inferring Activity from Observation
Person action sequences can be represented as a simple finite state machine.
Face tracker captures the (x,y) positions of faces in the image over time.
Global meeting state is inferred from aggregate of person activity.
Inferred state from classifier
•State transitions are encoded as a dynamic Bayesian network in a HMM structure.
•Current person state is a function of observed human activity and previous state.
![Page 10: CAMEO: Meeting Understanding Prof. Manuela M. Veloso, Prof. Takeo Kanade Dr. Paul E. Rybski, Dr. Fernando de la Torre, Dr. Brett Browning, Raju Patil,](https://reader036.vdocuments.net/reader036/viewer/2022062409/5697bf7d1a28abf838c84b4b/html5/thumbnails/10.jpg)
Logging/Replay/Towards Learning
Tracked person data is recorded for off-line activity analysis and learning of dynamics.
The recorded logs can be replayed back through CAMEO.
Model-based simulation generates high-level state descriptions of group activies.
Data-based simulation generates low-level “frame-by-frame” individual person activity state descriptions.
bring [carlos, computer]bring [carlos, cameo]set_up [carlos, computer]use [carlos, computer]set_up [carlos, cameo]use [carlos, cameo]give_demo [carlos, face_recognition]ask_question [fernando, face_recognition]answer_question [carlos, fernando, face_recognition]give_demo [raju, tracking]ask_question [carlos, tracking]answer_question [raju, carlos, tracking]give_demo [carlos, face_detection]ask_question [raju, face_detection]answer_question [carlos, raju, face_detection]ask_question [fernando, face_detection]answer_question [raju, fernando, face_detection]remove [carlos, computer]remove [carlos, cameo]leave [jon]leave [raju]leave [fernando]leave [carlos]leave [daniel]