monkey see, monkey do, monkey… talk? by helen zou july 23, 2010
DESCRIPTION
Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010. Using Vocalization Features to Identify Ethanol Intoxication in Rhesus Macaques by Helen Zou July 23, 2010. Introduction Background Rhesus Macaques Speech processing Literature review Previous findings in humans - PowerPoint PPT PresentationTRANSCRIPT
Page 1
Monkey See, Monkey Do, Monkey… Talk?by Helen ZouJuly 23, 2010
Monkey See, Monkey Do, Monkey… Talk?by Helen ZouJuly 23, 2010
Page 2
Using Vocalization Features to Identify Ethanol Intoxication in Rhesus Macaquesby Helen ZouJuly 23, 2010
Using Vocalization Features to Identify Ethanol Intoxication in Rhesus Macaquesby Helen ZouJuly 23, 2010
Page 3
OverviewOverview
• Introduction • Background
– Rhesus Macaques– Speech processing
• Literature review– Previous findings in humans– Macaque vocalizations
• Experiment procedure• Data analysis
– Segmentation and clustering– Extracting features
• Results• Acknowledgments
• Introduction • Background
– Rhesus Macaques– Speech processing
• Literature review– Previous findings in humans– Macaque vocalizations
• Experiment procedure• Data analysis
– Segmentation and clustering– Extracting features
• Results• Acknowledgments
Page 4
IntroductionIntroduction
• Duke University – Class of 2013• Biomedical Engineering major and Neuroscience
minor• Emailed Dr. Grant because of her work with
primates and neuroscience• Vocalization project• Worked at both ONPRC and OGI• Not under any specific program, except…
• Had to give a presentation anyway
• Duke University – Class of 2013• Biomedical Engineering major and Neuroscience
minor• Emailed Dr. Grant because of her work with
primates and neuroscience• Vocalization project• Worked at both ONPRC and OGI• Not under any specific program, except…
• Had to give a presentation anyway
Page 5
Background – Rhesus MacaquesBackground – Rhesus Macaques
• Alcohol drug discrimination and self-administration
• Predictors of heavy drinking (dominance-related?)
• BEC (Blood Ethanol Concentration)• Need simpler way to measure intoxication in
social settings• Why not look at speech?
• Alcohol drug discrimination and self-administration
• Predictors of heavy drinking (dominance-related?)
• BEC (Blood Ethanol Concentration)• Need simpler way to measure intoxication in
social settings• Why not look at speech?
Page 6
Background – Speech ProcessingBackground – Speech Processing
• Voiced, unvoiced, and noise
• For monkeys, we focused on voiced (coos and screams)
• Potential features– Frequency and pitch– Shimmer (amplitude) and jitter (pitch)– Spectral entropy– Root mean square (energy)
• Voiced, unvoiced, and noise
• For monkeys, we focused on voiced (coos and screams)
• Potential features– Frequency and pitch– Shimmer (amplitude) and jitter (pitch)– Spectral entropy– Root mean square (energy)
Page 7
Sample Wave FormSample Wave Form
Page 8
Sample Voiced RegionSample Voiced Region
Page 9
Sample NoiseSample Noise
Page 10
Sample BackgroundSample Background
Page 11
OverviewOverview
• Introduction • Background
– Rhesus Macaques– Speech processing
• Literature review– Previous findings in humans– Exxon Valdez case– Macaque vocalizations
• Introduction • Background
– Rhesus Macaques– Speech processing
• Literature review– Previous findings in humans– Exxon Valdez case– Macaque vocalizations
Page 12
Prior Studies – KlingholzRecognition of low-level alcohol intoxication from speech signal (1988)
Prior Studies – KlingholzRecognition of low-level alcohol intoxication from speech signal (1988)
• Approach recognition of intoxication as speaker identification task
• Measure laryngeal and articulatory features– Laryngeal - fundamental frequency and
signal-to-noise (SNR)– Articulatory – formants (F1/F2 ratio)
• Major findings– Increased FO variation– Decreased SNR– Did not change F1/F2
• Limitation: small sample size• Much more accurate than human recognition
• Approach recognition of intoxication as speaker identification task
• Measure laryngeal and articulatory features– Laryngeal - fundamental frequency and
signal-to-noise (SNR)– Articulatory – formants (F1/F2 ratio)
• Major findings– Increased FO variation– Decreased SNR– Did not change F1/F2
• Limitation: small sample size• Much more accurate than human recognition
Page 13
Prior Studies – HollienEffects of ethanol intoxication on speech suprasegmentals (2001)
Prior Studies – HollienEffects of ethanol intoxication on speech suprasegmentals (2001)
• Measured several different features– Nonfluency increase is best measure– F0 increases and utterance duration increases
(moderate measure)– F0 variability slightly increases (poor measure)– Vocal intensity had no change
• 20% of subjects exhibited no consistent changes• Unfortunately, disagrees with the previous
findings
• Measured several different features– Nonfluency increase is best measure– F0 increases and utterance duration increases
(moderate measure)– F0 variability slightly increases (poor measure)– Vocal intensity had no change
• 20% of subjects exhibited no consistent changes• Unfortunately, disagrees with the previous
findings
Page 14
Exxon Valdez Court CaseAcoustic Analysis of Voice Recordings from the Exxon Valdez by J. Tanford et al (1992)
Exxon Valdez Court CaseAcoustic Analysis of Voice Recordings from the Exxon Valdez by J. Tanford et al (1992)
• Oil tanker crashed in Alaska in 1989
• Captain of ship denied intoxication
• Analysis of speech found:– Misspoken words– Slurred pronunciations– Slower speaking rate– Lower pitch– Increased f0 variability
• Characteristics were consistent with intoxication
• Oil tanker crashed in Alaska in 1989
• Captain of ship denied intoxication
• Analysis of speech found:– Misspoken words– Slurred pronunciations– Slower speaking rate– Lower pitch– Increased f0 variability
• Characteristics were consistent with intoxication
Page 15
Previous Study – WeertsPrimate vocalizations during social separation and aggression: effects of alcohol and benzodiazepines (1996)
Previous Study – WeertsPrimate vocalizations during social separation and aggression: effects of alcohol and benzodiazepines (1996)
• Focused on testing the effect of different social situations– Social separation: EtOH reduced isolation peeps– Aggression: EtOH increased aggression peeps
• Social context determines effect of drugs (potential confounding variable?)
• Focused on testing the effect of different social situations– Social separation: EtOH reduced isolation peeps– Aggression: EtOH increased aggression peeps
• Social context determines effect of drugs (potential confounding variable?)
Page 16
Summary of Previous WorkSummary of Previous Work
• Experiments done on the effect of intoxication on human speech have inconsistent findings
• Very few studies actually done on macaque vocalizations
• Many uncontrolled variables (long-term voice effort, social context, etc.)
• Definitely some effect of ethanol intoxication on speech features
• Experiments done on the effect of intoxication on human speech have inconsistent findings
• Very few studies actually done on macaque vocalizations
• Many uncontrolled variables (long-term voice effort, social context, etc.)
• Definitely some effect of ethanol intoxication on speech features
Page 17
The QuestionThe Question
• Will the vocalizations of monkeys change when intoxicated versus when sober?
• Will the vocalizations of monkeys change when intoxicated versus when sober?
Page 18
MethodsMethods
• Put recorders on the monkeys• Gavage with water or alcohol (alternating)• Measure BECs in one hour• Take off recorders• Analyze data for various features• Identify differences in vocalization• Draw conclusions from data and voila!
• But in reality…
• Put recorders on the monkeys• Gavage with water or alcohol (alternating)• Measure BECs in one hour• Take off recorders• Analyze data for various features• Identify differences in vocalization• Draw conclusions from data and voila!
• But in reality…
Page 19
ProblemsProblems
1. Exceeding recorder threshold
2. Not enough vocalizations
1. Exceeding recorder threshold
2. Not enough vocalizations
1. Attenuate with rubber and foam
2. Switch to more vocal monkey
1. Attenuate with rubber and foam
2. Switch to more vocal monkey
SolutionsSolutions
Page 20
Clementine Example WaveformClementine Example Waveform
Page 21
Data Analysis?Data Analysis?
• Recordings had vocalizations, noise, silence, other monkeys, etc.
• How would we isolate the monkey of interest?
• Recordings had vocalizations, noise, silence, other monkeys, etc.
• How would we isolate the monkey of interest?
Page 22
Sample SpectrumSample Spectrum
NoiseNoise
VocalizationsVocalizations
Page 23
Clementine Example SpectrumClementine Example Spectrum
Page 24
Data AnalysisData Analysis
1. Cut the wave file into smaller segments2. Isolate vocalization parts of speech3. Extract features for vocalization regions4. Compare features for intoxicated versus sober
speech
1. Cut the wave file into smaller segments2. Isolate vocalization parts of speech3. Extract features for vocalization regions4. Compare features for intoxicated versus sober
speech
Page 25
Segmentation/ClusteringRobust Speaker Change Detection by J. Ajmera et al. (2003)
Segmentation/ClusteringRobust Speaker Change Detection by J. Ajmera et al. (2003)
• Originally created for separating speakers in news broadcasts
• Find likely change points• Segment data with overlapping frames• Cluster similar segments (by speaker)
• Originally created for separating speakers in news broadcasts
• Find likely change points• Segment data with overlapping frames• Cluster similar segments (by speaker)
Page 26
Segmentation and ClusteringSegmentation and Clustering
Page 27
Data AnalysisData Analysis
1. Cut the wave file into smaller segments2. Isolate vocalization parts of speech3. Extract features for vocalization regions4. Compare features for intoxicated versus sober
speech
1. Cut the wave file into smaller segments2. Isolate vocalization parts of speech3. Extract features for vocalization regions4. Compare features for intoxicated versus sober
speech
Page 28
Spectrum – Human vs. MonkeySpectrum – Human vs. Monkey
Page 29
Results – Human vs. MonkeyResults – Human vs. Monkey
• Bandwidth of formants in monkey vocalizations is larger than for humans
• Humans have more formants (5+), monkeys have much fewer (2-4)
• Distance between the formants for monkeys is much larger than between human formants
• Shape of formants is curved for screams and straight for coos
• Bandwidth of formants in monkey vocalizations is larger than for humans
• Humans have more formants (5+), monkeys have much fewer (2-4)
• Distance between the formants for monkeys is much larger than between human formants
• Shape of formants is curved for screams and straight for coos
Page 30
Spectrum – Human vs. MonkeySpectrum – Human vs. Monkey
Human Noise Coo Scream
Page 31
Results - F0 graphsResults - F0 graphs
Page 32
Results – Alcohol vs. WaterResults – Alcohol vs. Water
• Graphed all of the features
• F0 as x-variable produced most significant results
• F0 tends to be higher during intoxication
• Graphed all of the features
• F0 as x-variable produced most significant results
• F0 tends to be higher during intoxication
Page 33
Results – Rms vs. f0Results – Rms vs. f0
Page 34
ResultsResults
• Root mean square (energy) vs. fundamental frequency
• Control vocalizations have larger variation in energy
• Intoxication has higher f0
• Root mean square (energy) vs. fundamental frequency
• Control vocalizations have larger variation in energy
• Intoxication has higher f0
Page 35
Results – Rms vs. Spec entropyResults – Rms vs. Spec entropy
Page 36
ResultsResults
• Spectral entropy vs. f0
• Control vocalizations have larger variation in spectral entropy
• Intoxication has higher f0
• Spectral entropy vs. f0
• Control vocalizations have larger variation in spectral entropy
• Intoxication has higher f0
Page 37
Results Results
• Alcohol increases fundamental frequency (agrees with Hollien study)
• Alcohol decreases variation in energy and spectral entropy
• Consistent with alcohol impairing muscle control of vocal cords
• Alcohol increases fundamental frequency (agrees with Hollien study)
• Alcohol decreases variation in energy and spectral entropy
• Consistent with alcohol impairing muscle control of vocal cords
Page 38
LimitationsLimitations
• Very small sample size• Limited number of vocalizations• Lots of silence and noise in
recordings• BEC was low (between .017
and .044)• Monkeys were separated – may
have different results in social setting
• Only paired comparisons
• Very small sample size• Limited number of vocalizations• Lots of silence and noise in
recordings• BEC was low (between .017
and .044)• Monkeys were separated – may
have different results in social setting
• Only paired comparisons
Page 39
In the FutureIn the Future
• Further study correlations between different vocalization features and intoxication
• Use recordings to correlate with other factors (such as stress, dominance, etc.)
• Find ways to increase vocalizations• Pair vocal recordings with visual tracking • Measure ethanol intake using vocalizations in
social settings• Expand studies to other species
• Further study correlations between different vocalization features and intoxication
• Use recordings to correlate with other factors (such as stress, dominance, etc.)
• Find ways to increase vocalizations• Pair vocal recordings with visual tracking • Measure ethanol intake using vocalizations in
social settings• Expand studies to other species
Page 40
ConclusionConclusion
• Added to the studies done on macaque vocalizations
• Used computer algorithms to separate and analyze data
• Found that formants are a good way to separate human and monkey vocalizations
• Alcohol increases f0 and decreases variability of energy and spectral entropy
• Eventually use vocalizations to measure intoxication in macaques in social settings
• Added to the studies done on macaque vocalizations
• Used computer algorithms to separate and analyze data
• Found that formants are a good way to separate human and monkey vocalizations
• Alcohol increases f0 and decreases variability of energy and spectral entropy
• Eventually use vocalizations to measure intoxication in macaques in social settings
Page 41
AcknowledgmentsAcknowledgments
• Dr. Kathy Grant• Dr. Izhak Shafran• The Grant Lab (Kevin Nusser, Andrew Rau,
Jessica Shaw, and Cara Candell)• Meysam Asgari• OGI and ONPRC staff and coworkers
• Dr. Kathy Grant• Dr. Izhak Shafran• The Grant Lab (Kevin Nusser, Andrew Rau,
Jessica Shaw, and Cara Candell)• Meysam Asgari• OGI and ONPRC staff and coworkers
Page 42
Questions?Questions?
Page 43
Prior Studies - KlingholzPrior Studies - Klingholz
• Approach recognition of intoxication as speaker identification task
• 11 human test subjects and 5 controls• Read a text segment in German• Measure laryngeal and articulatory features
– Laryngeal - fundamental frequency and signal-to-noise (SNR)
– Articulatory – formants (F1/F2 ratio)• Intoxication results
– Increased FO variation– Decreased SNR– Did not change F1/F2
• Correlation between BAL and F0• Long-term voice effort has similar effect• Much more accurate than human recognition
• Approach recognition of intoxication as speaker identification task
• 11 human test subjects and 5 controls• Read a text segment in German• Measure laryngeal and articulatory features
– Laryngeal - fundamental frequency and signal-to-noise (SNR)
– Articulatory – formants (F1/F2 ratio)• Intoxication results
– Increased FO variation– Decreased SNR– Did not change F1/F2
• Correlation between BAL and F0• Long-term voice effort has similar effect• Much more accurate than human recognition
Page 44
Prior Studies - HollienPrior Studies - Hollien
• Speech samples at four levels of intoxication• 35 human subjects• Results
– Nonfluency increase is best measure– F0 increases and utterance duration increases
(moderate measure)– F0 variability increases (poor measure)– Vocal intensity had no change
• 20% of subjects exhibited no consistent changes
• Speech samples at four levels of intoxication• 35 human subjects• Results
– Nonfluency increase is best measure– F0 increases and utterance duration increases
(moderate measure)– F0 variability increases (poor measure)– Vocal intensity had no change
• 20% of subjects exhibited no consistent changes
Page 45
Prior Studies - WeertsPrior Studies - Weerts
• 33 squirrel monkeys in two different social situations
• Social separation: EtOH reduced isolation peeps• Aggression: EtOH increased aggression peeps• Social context determines effect of drugs
• 33 squirrel monkeys in two different social situations
• Social separation: EtOH reduced isolation peeps• Aggression: EtOH increased aggression peeps• Social context determines effect of drugs