signals relevant to counter-terror
DESCRIPTION
- PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/1.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Computer vision approaches to identifying Computer vision approaches to identifying people and possible malfeasant behavior. people and possible malfeasant behavior.
Dimitris N. Metaxas Mark G. FrankDimitris N. Metaxas Mark G. Frank Director, Computational Biomedicine School of Communication, Imaging & Modeling Center Information & Library Studies
with special thanks to: Paul Ekman, UCSFUCSF;Sinuk Kang, Amy Marie Keller, Anastacia Kurylo, Maggie Herbasz, Belida Uckun, RutgersRutgers;Jeff Cohn, PittPitt; Takeo Kanade, CMUCMU; Javier Movellan & Marni Bartlett, UCSDUCSD.David Dinges, UPENN
Also thanks to: Office of Naval Research, National Science Foundation (ITR program), AFOSR, DARPA
![Page 2: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/2.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
![Page 3: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/3.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Signals relevant to counter-terror.Signals relevant to counter-terror.
• Identification of bad guysIdentification of bad guys• Changes in gait with loads as small as 1 kgChanges in gait with loads as small as 1 kg• Anger prior to imminent attackAnger prior to imminent attack• Fear/distress when lyingFear/distress when lying
![Page 4: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/4.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Take a closer look…Take a closer look…
Kim Philby, 1960’sKim Philby, 1960’s (Frank & Ekman, 2003)(Frank & Ekman, 2003)
![Page 5: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/5.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
How lies are betrayed.How lies are betrayed.
LieLie
Cognitive cluesCognitive clues
- Contradictory statements- Contradictory statements- HesitationsHesitations
-Speech errorsSpeech errors-Reduced illustratorsReduced illustrators
-Contradictory emblemsContradictory emblems-Reduced detailReduced detail
-Etc.Etc.
Emotional cluesEmotional clues
Lying about feelingsLying about feelings Feelings about lyingFeelings about lying
-Look for -Look for reliablereliable
signs of signs of emotionemotion
-Duping delight-Duping delight-GuiltGuilt
-DetectionDetectionapprehensionapprehension
![Page 6: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/6.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
The Facial Action Coding System The Facial Action Coding System (FACS)(FACS)
Ekman & Friesen, 1978
Action code: 1, 2, 4, 5, 7, 20, 26
1 Inner brow raise2 Outer brow raise4 Brow lower5 Upper lid raise7 Lid tighten20 Lip stretch26 Jaw drop
46 Action Units
![Page 7: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/7.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Challenges facing behavioral science:Challenges facing behavioral science:
• Advantages:Advantages:- reliably identify people & behaviors - non-obtrusive- non-inferential, allows for discovery
• Disadvantage:Disadvantage:- laborious - mistaken identity via cognitive capacity, disguise, etc
• SolutionSolution- automatic computer vision techniques
![Page 8: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/8.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Gait recognition
• Identify people from the way they walk
• Important for surveillance and intrusion detection
• What are good features for identifying a person?– i.e., what features are person-specific?
![Page 9: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/9.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Background
Sagittal plane - divides body into left and right halves
Limb segment - a vector between two sites on a particular limb
![Page 10: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/10.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Elevation Angles
![Page 11: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/11.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
The trajectories of the sagittal elevation angles are invariant across different subjects.
As a consequence, person-independent gait recognition will require less training data.
(Borghese et al., 1996)
![Page 12: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/12.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
The cyclogram
• Elevation angles trace curve in a 4D space
• Curve is called “cyclogram”
• Cyclogram lies in a 2D plane– Well, almost
• Hypothesis: deviation of cyclogram from plane is person-specific
![Page 13: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/13.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Cyclogram example
Curve is cyclogram projected into best-fit plane
Green points are real points of cyclogram
Red lines trace the deviation of points from plane (exaggerated scale)
![Page 14: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/14.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Cyclogram sequence
• Deviation from cyclogram plane can be represented as a sequence
• e.g., CCCGTTTTATATTTTTAAAAGCCGGTAAATTAGGGG
• Compare sequences between people via longest common subsequence (LCS) matching– Well-known dynamic programming algorithm,
used in computational biology
![Page 15: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/15.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Examples of People Detection
![Page 16: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/16.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Examples of Gait Analysis
![Page 17: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/17.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Face: Tracking: Stress Recognition
• Identify which Facial Features (space and time) are important to recognize stress
• Assymetries and Movements around the mouth and eyebrows
![Page 18: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/18.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Slope = Asymmetry
A horizontal line would indicate no asymmetry. This facial expression, however, is generally slanted upward and to the left.
![Page 19: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/19.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Plots of high and low stress
Expression of high stress in form ofasymmetrical facial expression
time
asym
met
ry
![Page 20: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/20.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
In contrast, low stress
time
asym
met
ry
![Page 21: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/21.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Some more high stress from different subjects
![Page 22: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/22.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Face: Tracking: Stress Recognition
![Page 23: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/23.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Face: Tracking: Stress Recognition
![Page 24: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/24.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
![Page 25: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/25.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Subtle brow changes important.Subtle brow changes important.
![Page 26: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/26.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Technical Challenges.Technical Challenges.
• PosePose• Head motionHead motion• Occlusion from glasses, facial hair, rotation, Occlusion from glasses, facial hair, rotation,
handshands• TalkingTalking• Video qualityVideo quality• Frame rate (blinks)Frame rate (blinks)
![Page 27: Signals relevant to counter-terror](https://reader036.vdocuments.net/reader036/viewer/2022062322/5681520f550346895dc05219/html5/thumbnails/27.jpg)
M. Frank & D. Metaxas - Rutgers HMS Symposium 2003
Conclusions.Conclusions.
• There are reliable means to identify people as There are reliable means to identify people as well as behaviors associated with deception and well as behaviors associated with deception and hostile intent.hostile intent.
• We can detect these behaviors. We can detect these behaviors.
• We can represent them digitally.We can represent them digitally.
• Can this make us more secure?Can this make us more secure?