detection and computational analysis of psychological signals … · detection and computational...
TRANSCRIPT
Detection and Computational Analysisof Psychological Signals (DCAPS)
Daniel “Rags” Ragsdale, PhD
Program Manager
National Initiative for Arts & Health in the Military’s Third National Summit: “Advancing Research in the Arts for Health and Well-being across the
Briefing for:
Advancing Research in the Arts for Health and Well being across the Military Continuum”
February 27, 2015
1Distribution Statement A - Approved for Public Release, Distribution Unlimited
DCAPS Goal
• Develop computationally-based, networked tools to assess and analyze psychological behavioral states in a DoD medical environment.
• Analyze verbal and nonverbal cues, online activities, and indicators of social behavior
• Employ approaches both individually and in concertEmploy approaches both individually and in concert
• Use novel sources of data: iPhone, Android, Kinect, social-networks, synthetic environments, EEGs, web cams, etc.cams, etc.
• Leverage expertise from MIT, Dartmouth, Boston University, Vanderbilt, USC, & UCLA
2Distribution Statement A - Approved for Public Release, Distribution Unlimited
DCAPS: BLUF
High-level contributions:
• Early recognition of indicators of psychological distress (e g depression anxiety or PTSD)(e.g., depression, anxiety or PTSD)
• More effective triage of potentially distressed service members
• Facilitate clinician trainingg
• Share DCAPS insights and data with scientific community
• Virtual humans may reduce stress and fear of judgment
Specific Applications:
• Behavioral assessment and monitoring of military personnel, pre-during and post deployment, during, and post-deployment
• Deployed to clinician training sites
• Call Centers/HotlinesCall Centers/Hotlines
3Distribution Statement A - Approved for Public Release, Distribution Unlimited
Used “Honest Signal” theory as a start point (Dr Sandy Pentland MIT)
DCAPS Approach
• Used “Honest Signal” theory as a start point (Dr. Sandy Pentland, MIT)
• Data Analyzed
• Spoken and written communications (including metadata)
• Online behaviors: email, surfing habits, games, and social networking
• Nonverbal cues such as facial expression, posture, movement patterns
• Creative content of artistic expression in art and writing (SBIR)p g ( )
• Response to stimuli obtained through sensors such as EEG, EKG, GSR
• “Products”
R lit A l i t li ti th t th “h t i l ” l i• Reality Analysis: two applications that use the “honest signals” analysis
• Clinician (TRL 9) and Mobile (TRL 6)
• MINAT: text and voice analytics tools for early detection of PTSD, depression, and suicidality (TRL 6)
• SimSensei : an avatar-based tool to perform “honest signal” analysis that examines a combination of nonverbal cues using Kinect technologies (TRL 4)
4Distribution Statement A - Approved for Public Release, Distribution Unlimited
REALITY ANALYSIS
Mobile
Clinician
55Distribution Statement A - Approved for Public Release, Distribution Unlimited
Reality Analysis Mobile: Overall Description
• Goal: Using data from mobile phone usage provide individuals (and clinicians) by with a high-level assessment of an individual’s mental health state
d d l h h h d• Individuals have the option to share data
• Data analyzed:
• Activity / location
• Sleep patterns
• Audio Diary content
• Call and text metadata• Call and text metadata
• Call and text content (when used with MINAT)
• Analysis produced are qualitative assessments in four areas:
• Travel, Social, Sleep, Mood
6Distribution Statement A - Approved for Public Release, Distribution Unlimited
Reality Analysis Mobile: Results
• Participants’ behavioral and voice data were passively gathered then assessed using predictive models
• Assessments were completed using data from clinical and non-clinical field trials*
• Clinical trial• Clinical trial
• 95 participants over a 3-month period, included weekly audio diary entries • Trial included a 30K weekly survey questions and over 51M data points of bio-
behavioral markers
• Protocol adherence was 96%
• Non-Clinical trial • Two focus groups, totaling 21 participants• Participants were comfortable with the level of privacy of the mobile app
• Participants were also comfortable sharing their mental health status
• Gather data on adoption, usability, and privacy concerns
• The predictive models produced 0.6 to 0.9 AUC (ROC curves) for a variety of symptoms [1]
7
[1] Feast, J. (2014 , February). Cogito & DCAPS, Research project review presented at PM Site Review Meeting, Cambridge, MA.
Distribution Statement A - Approved for Public Release, Distribution Unlimited
Reality Analysis Mobile: Predictive Model Results[1]
S tSymptoms
88
[1] Feast, J. (2014 , February). Cogito & DCAPS, Research project review presented at PM Site Review Meeting, Cambridge, MA.
Distribution Statement A - Approved for Public Release, Distribution Unlimited
Reality Analysis Mobile: Privacy Ratings [1]
9
[1] Place, S. (2014)Cogito Health DCAPS Final Report, Cambridge, MA.
9Distribution Statement A - Approved for Public Release, Distribution Unlimited
Reality Analysis Mobile: Data Shared with Other Entities/Groups [1]
1010
[1] Place, S. (2014)Cogito Health DCAPS Final Report, Cambridge, MA.
Distribution Statement A - Approved for Public Release, Distribution Unlimited
Reality Analysis Clinician: Overall Description
• Goal: Analyze verbal interactions between clinician and patient to infer:
1 2
• Degree of engagement
• Empathy,
• Active listing
3
g
• Graphical representations:
• Tone maps
C l b l b
4
Di l Di l• Conversational balance bar
• Conversational flow gauge
• Speech pattern timeline
Dialog Display: (1) Tone Maps plots the distribution of
dynamic variation and speaking rate(2) Conversational Balance Bar indicates
th l ti ti i ti f h k• Supports both real-time and
post analysis
the relative participation of each speaker(3) Conversational Flow Gauge measures
of the fluidity of the conversation(4) Speech Pattern Timeline captures the
tt f h d i th ll
11
pattern of speech during the call.
Distribution Statement A - Approved for Public Release, Distribution Unlimited
Reality Analysis Clinician: Real-time Interface [1]
Clinician Desktop
1212
[1] Rizzo, A., Morency, L.P., and Gratch J. (December 2013) Detection and Computational Analysis of Psychological Signals: TeleCoach Final Report, Final Technical Report, Playa Vista, CA.
Distribution Statement A - Approved for Public Release, Distribution Unlimited
MINATMINAT(Medical Informatics and Analytics Toolkit)
1313Distribution Statement A - Approved for Public Release, Distribution Unlimited
MINAT: Overall Description
• Goal: Assign a patient’s communication into zero, one, or more “codes” that are potential indicators for PTSD, depression, & suicidality
A i t ith t i• Assist with triage
• Communication with the patient performed using transcribed audio and written text
• Analysis produced is zero, one, or more “indicators”
• Categories of indicators:
• Stress Exposure• Affect• Behavior• Cognitive state• Degrees of Impairment
• Runs as a deployable web servicep y
14Distribution Statement A - Approved for Public Release, Distribution Unlimited
MINAT: Indicator Codes
• 70 indicator codes in five categories
• Categories of indicator codes
• Behaviors• Behaviors
• Stress Exposure
• Affect
• Cognitive states• Cognitive states
• Degrees of Impairment
DSM IV i i d• DSM-IV inspired
• Criteria, symptoms, indicators
COLING 2012
• Indicator codes continually refined with experience
• Reduces ambiguity
• Increases inter-annotator agreement
• Supports machine learning15Distribution Statement A - Approved for Public Release, Distribution Unlimited
MINAT: Indicator Detection
1616Distribution Statement A - Approved for Public Release, Distribution Unlimited
SIMSENSEI
MultiSense / TeleCoach
1717Distribution Statement A - Approved for Public Release, Distribution Unlimited
SimSensei: Overall Description
• Goal: Create an avatar-based tool that senses nonverbal cues that are potential indicators of mental distress
• Communicates with an individual via conversationalCommunicates with an individual via conversational
interaction
• Data analyzed:y
• Body movement
• Gestures & other physical movements
• Facial displays
• Voice
• Response latency
• Real-time analysis produced: SimSensei: Ellie
18
• Horizontal gaze, smile level, voice, attention, body movement
Distribution Statement A - Approved for Public Release, Distribution Unlimited
SimSensei / Telecoach Results
• Goal: Assess the clinical states of interest through a multimodal analysis of a patient’s verbal and nonverbal communication
• Communication with the patient is with clinician through shared video• Communication with the patient is with clinician, through shared video
• Training
• 300+ 20-minute interviews with Ellie
• Participants’ behavioral profiles were indicative of PTSD and depression
• Demonstrated potential as an alternative clinician support tool
19Distribution Statement A - Approved for Public Release, Distribution Unlimited
MultiSense Overlay: Potential Real-Time Display
20
*Person in image is a veteran participant, who has consented to sharing his image.
Distribution Statement A - Approved for Public Release, Distribution Unlimited
Psychological Distress Indicators (IEEE FG 2013, Interspeech 2013, MMVR 2013)
Vertical eye gaze Smile intensityJoy – Facial expr. Sad – Facial expr.
Distress
Hand self-adaptor Legs fidgeting
Distress No-distressNo-distress
Voice energy std. Voice quality
Distress Distress No-distressNo-distressDistress
AssessmentInterviewCorpusCorpus (DAIC)
Distress No-distress Distress Distress No-distressNo-distressDistress No-distress
21Distribution Statement A - Approved for Public Release, Distribution Unlimited
TeleCoach: Prediction Results
• Individuals are able to incorporate the TeleCoach indicators into their assessments of a patient’s mental distress.
• Specifically, providing individuals with TeleCoach indicators cues improved the accuracy of their assessments
Without indicators With indicators
Accuracy of depression ß = 0.59 ß = 0.72y pratings
Accuracy of PTSD ratings ß = 0.17 ß = 0.43
ß represents the size of the effect, or in this case, degree of accuracy, in regression analyses. ß ranges from 0.0, which would be no accuracy, to 1.0, which represents perfect accuracy.
22Distribution Statement A - Approved for Public Release, Distribution Unlimited
DCAPS: Bibliography of refereed published papers
• Jon Gratch, Louis-Philippe Morency, Stefan Scherer, Giota Stratou, Jill Boberg, Sebastian Koenig, Todd Adamson and Albert Rizzo. “User-State Sensing for Virtual Health Agents and TeleHealth Applications,” the NextMed/MMVR20 conference, San Diego, CA, February 20, 2013.
• Stefan Scherer, Giota Stratou, Marwa Mahmoud, Jill Boberg, Jonathan Gratch, Albert (Skip) Rizzo, Louis-Philippe Morency, “Automatic Behavior Descriptors for Psychological Disorder Analysis”, 10th IEEE International Conference on Automatic Face and Gesture Recognition, April 2013.
• Sunghyun Park and Louis-Philippe Morency, “Crowdsourcing Micro-Level Multimedia Annotations: The Challenges of Evaluation and Interface,” ACM Multimedia Workshop on Multimedia Crowdsourcing, 2012.
St f S h St M ll Gi t St t Y X F b i i M bi i Al i E Alb t• Stefan Scherer, Stacy Marsella, Giota Stratou, Yuyu Xu, Fabrizio Morbini, Alesia Egan, Albert (Skip) Rizzo and Louis-Philippe Morency, “Perception Markup Language: Towards a Standardized Representation of Perceived Nonverbal Behaviors,” Intelligent Virtual Agent Conference (IVA), 2012.
• E. Suma, B. Lange, A. Rizzo, D. Krum, and M. Bolas. “FAAST-R: Defining a core mechanic for designing gestural interfaces.” In The 3rd Dimension of CHI: Touching and Designing 3D User Interfaces, 2012. Accepted for publication.
• Fabrizio Morbini, Eric Forbell, David DeVault, Kenji Sagae, David Traum and Albert Rizzo. “A ab o o b , c o be , a d e au t, e j Sagae, a d au a d be t oMixed-Initiative Conversational Dialogue System for Healthcare”. Demonstration, in Proceedings of the SIGDIAL 2012 Conference.
23Distribution Statement A - Approved for Public Release, Distribution Unlimited
DCAPS: Bibliography of refereed published papers (cont.)
• N. Burba, M. Bolas, D. Krum, and E. Suma. “Unobtrusive measurement of subtle nonverbal behaviors with the Microsoft Kinect.” In IEEE VR Workshop on Ambient Information Technologies, pages 10–13, 2012.
• Saleem S ; Prasad R Vitaladevuni S ; Pacula M ; Crystal M; Marx B ; Sloan D ;• Saleem, S.; Prasad, R., Vitaladevuni, S.; Pacula, M.; Crystal, M; Marx, B.; Sloan, D.; Vasterling, D.; Speroff, T.: “Automatic Detection of Psychological Distress Indicators from Online Forum Posts”, Proceedings of the 2012 International Conference on Computational Linguistics (COLING 2012), Mumbai, India.
A th k i h S V b A N P d R "M d l b d t i f t f ti• Ananthakrishnan, S.; Vembu, A.N.; Prasad, R., "Model-based parametric features for emotion recognition from speech," Automatic Speech Recognition and Understanding (ASRU), 2011 IEEE Workshop, vol., no., pp.529,534, 11-15 Dec. 2011.
• Rozgic, V.; Ananthakrishnan, S.; Saleem, S.; Kumar, R.; Prasad, R., "Ensemble of SVM trees gfor multimodal emotion recognition," Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific, vol., no., pp.1,4, 3-6 Dec. 2012.
• Saleem, S.; Pacula, M.; Chasin, R.; Kumar, R.; Prasad, R.; Crystal, M.; Marx, B.; Sloan, D.; Vasterling, J.; Speroff, T., "Automatic detection of psychological distress indicators in onlineVasterling, J.; Speroff, T., Automatic detection of psychological distress indicators in online forum posts," Signal & Information Processing Association Annual Summit and Conference (APSIPA ASC), 2012 Asia-Pacific, vol., no., pp.1,4, 3-6 Dec. 2012.
• Rozgic, V.; Ananthakrishnan, S.; Saleem, S.; Kumar, R.; Vembu, A.N.; Prasad, R. "Emotion Recognition using Acoustic and Lexical Features " Proceedings of Interspeech Portland ORRecognition using Acoustic and Lexical Features, Proceedings of Interspeech, Portland, OR 2012.
24Distribution Statement A - Approved for Public Release, Distribution Unlimited
Technology Readiness Levels
• TRL 1: Basic principles observed and reported
• TRL 2: Technology concept and/or application formulated
TRL 3: Analytical and experimental critical function and/or characteristic proof• TRL 3: Analytical and experimental critical function and/or characteristic proof of concept
• TRL 4: Component and/or breadboard validation in a laboratory environment
• TRL 5: Component and/or breadboard validation in a relevant environment
• TRL 6: System/subsystem model or prototype demonstration in a relevant environment
• TRL 7: System prototype demonstration in an operational environment
• TRL 8: Actual system completed and qualified through test and demonstrationdemonstration
• TRL 9: Actual system proven through successful mission operations.
27Distribution Statement A - Approved for Public Release, Distribution Unlimited
Triage Assessment (VA Suicide Hotline), ICASSP 2014
We can reliably detect severe distress in hotline conversations: 80% recall in most severe cases (TR1) with only 19% False Positive Rate
1
TR3: No Referral1
TR2: Refer to Clinician1
TR1: Imminent Danger
0.7
0.8
0.9
0.7
0.8
0.9
0.7
0.8
0.9
0.3
0.4
0.5
0.6
TPR
0.3
0.4
0.5
0.6
TPR
0.3
0.4
0.5
0.6
TPR
AUC: 0 81 AUC: 0 69 AUC: 0 90
0
0.1
0.2
0 0.5 1
0
0.1
0.2
0 0.5 1
0
0.1
0.2
0 0.5 1
AUC: 0.81 AUC: 0.69 AUC: 0.90
FPRFPRFPR
28Distribution Statement A - Approved for Public Release, Distribution Unlimited
Stress Exposure Labels
29Distribution Statement A - Approved for Public Release, Distribution Unlimited