monkey see, monkey do, monkey… talk? by helen zou july 23, 2010

45
Page 1 Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Upload: agatha

Post on 06-Jan-2016

53 views

Category:

Documents


6 download

DESCRIPTION

Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010. Using Vocalization Features to Identify Ethanol Intoxication in Rhesus Macaques by Helen Zou July 23, 2010. Introduction Background Rhesus Macaques Speech processing Literature review Previous findings in humans - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 1

Monkey See, Monkey Do, Monkey… Talk?by Helen ZouJuly 23, 2010

Monkey See, Monkey Do, Monkey… Talk?by Helen ZouJuly 23, 2010

Page 2: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 2

Using Vocalization Features to Identify Ethanol Intoxication in Rhesus Macaquesby Helen ZouJuly 23, 2010

Using Vocalization Features to Identify Ethanol Intoxication in Rhesus Macaquesby Helen ZouJuly 23, 2010

Page 3: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 3

OverviewOverview

• Introduction • Background

– Rhesus Macaques– Speech processing

• Literature review– Previous findings in humans– Macaque vocalizations

• Experiment procedure• Data analysis

– Segmentation and clustering– Extracting features

• Results• Acknowledgments

• Introduction • Background

– Rhesus Macaques– Speech processing

• Literature review– Previous findings in humans– Macaque vocalizations

• Experiment procedure• Data analysis

– Segmentation and clustering– Extracting features

• Results• Acknowledgments

Page 4: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 4

IntroductionIntroduction

• Duke University – Class of 2013• Biomedical Engineering major and Neuroscience

minor• Emailed Dr. Grant because of her work with

primates and neuroscience• Vocalization project• Worked at both ONPRC and OGI• Not under any specific program, except…

• Had to give a presentation anyway

• Duke University – Class of 2013• Biomedical Engineering major and Neuroscience

minor• Emailed Dr. Grant because of her work with

primates and neuroscience• Vocalization project• Worked at both ONPRC and OGI• Not under any specific program, except…

• Had to give a presentation anyway

Page 5: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 5

Background – Rhesus MacaquesBackground – Rhesus Macaques

• Alcohol drug discrimination and self-administration

• Predictors of heavy drinking (dominance-related?)

• BEC (Blood Ethanol Concentration)• Need simpler way to measure intoxication in

social settings• Why not look at speech?

• Alcohol drug discrimination and self-administration

• Predictors of heavy drinking (dominance-related?)

• BEC (Blood Ethanol Concentration)• Need simpler way to measure intoxication in

social settings• Why not look at speech?

Page 6: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 6

Background – Speech ProcessingBackground – Speech Processing

• Voiced, unvoiced, and noise

• For monkeys, we focused on voiced (coos and screams)

• Potential features– Frequency and pitch– Shimmer (amplitude) and jitter (pitch)– Spectral entropy– Root mean square (energy)

• Voiced, unvoiced, and noise

• For monkeys, we focused on voiced (coos and screams)

• Potential features– Frequency and pitch– Shimmer (amplitude) and jitter (pitch)– Spectral entropy– Root mean square (energy)

Page 7: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 7

Sample Wave FormSample Wave Form

Page 8: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 8

Sample Voiced RegionSample Voiced Region

Page 9: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 9

Sample NoiseSample Noise

Page 10: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 10

Sample BackgroundSample Background

Page 11: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 11

OverviewOverview

• Introduction • Background

– Rhesus Macaques– Speech processing

• Literature review– Previous findings in humans– Exxon Valdez case– Macaque vocalizations

• Introduction • Background

– Rhesus Macaques– Speech processing

• Literature review– Previous findings in humans– Exxon Valdez case– Macaque vocalizations

Page 12: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 12

Prior Studies – KlingholzRecognition of low-level alcohol intoxication from speech signal (1988)

Prior Studies – KlingholzRecognition of low-level alcohol intoxication from speech signal (1988)

• Approach recognition of intoxication as speaker identification task

• Measure laryngeal and articulatory features– Laryngeal - fundamental frequency and

signal-to-noise (SNR)– Articulatory – formants (F1/F2 ratio)

• Major findings– Increased FO variation– Decreased SNR– Did not change F1/F2

• Limitation: small sample size• Much more accurate than human recognition

• Approach recognition of intoxication as speaker identification task

• Measure laryngeal and articulatory features– Laryngeal - fundamental frequency and

signal-to-noise (SNR)– Articulatory – formants (F1/F2 ratio)

• Major findings– Increased FO variation– Decreased SNR– Did not change F1/F2

• Limitation: small sample size• Much more accurate than human recognition

Page 13: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 13

Prior Studies – HollienEffects of ethanol intoxication on speech suprasegmentals (2001)

Prior Studies – HollienEffects of ethanol intoxication on speech suprasegmentals (2001)

• Measured several different features– Nonfluency increase is best measure– F0 increases and utterance duration increases

(moderate measure)– F0 variability slightly increases (poor measure)– Vocal intensity had no change

• 20% of subjects exhibited no consistent changes• Unfortunately, disagrees with the previous

findings

• Measured several different features– Nonfluency increase is best measure– F0 increases and utterance duration increases

(moderate measure)– F0 variability slightly increases (poor measure)– Vocal intensity had no change

• 20% of subjects exhibited no consistent changes• Unfortunately, disagrees with the previous

findings

Page 14: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 14

Exxon Valdez Court CaseAcoustic Analysis of Voice Recordings from the Exxon Valdez by J. Tanford et al (1992)

Exxon Valdez Court CaseAcoustic Analysis of Voice Recordings from the Exxon Valdez by J. Tanford et al (1992)

• Oil tanker crashed in Alaska in 1989

• Captain of ship denied intoxication

• Analysis of speech found:– Misspoken words– Slurred pronunciations– Slower speaking rate– Lower pitch– Increased f0 variability

• Characteristics were consistent with intoxication

• Oil tanker crashed in Alaska in 1989

• Captain of ship denied intoxication

• Analysis of speech found:– Misspoken words– Slurred pronunciations– Slower speaking rate– Lower pitch– Increased f0 variability

• Characteristics were consistent with intoxication

Page 15: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 15

Previous Study – WeertsPrimate vocalizations during social separation and aggression: effects of alcohol and benzodiazepines (1996)

Previous Study – WeertsPrimate vocalizations during social separation and aggression: effects of alcohol and benzodiazepines (1996)

• Focused on testing the effect of different social situations– Social separation: EtOH reduced isolation peeps– Aggression: EtOH increased aggression peeps

• Social context determines effect of drugs (potential confounding variable?)

• Focused on testing the effect of different social situations– Social separation: EtOH reduced isolation peeps– Aggression: EtOH increased aggression peeps

• Social context determines effect of drugs (potential confounding variable?)

Page 16: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 16

Summary of Previous WorkSummary of Previous Work

• Experiments done on the effect of intoxication on human speech have inconsistent findings

• Very few studies actually done on macaque vocalizations

• Many uncontrolled variables (long-term voice effort, social context, etc.)

• Definitely some effect of ethanol intoxication on speech features

• Experiments done on the effect of intoxication on human speech have inconsistent findings

• Very few studies actually done on macaque vocalizations

• Many uncontrolled variables (long-term voice effort, social context, etc.)

• Definitely some effect of ethanol intoxication on speech features

Page 17: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 17

The QuestionThe Question

• Will the vocalizations of monkeys change when intoxicated versus when sober?

• Will the vocalizations of monkeys change when intoxicated versus when sober?

Page 18: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 18

MethodsMethods

• Put recorders on the monkeys• Gavage with water or alcohol (alternating)• Measure BECs in one hour• Take off recorders• Analyze data for various features• Identify differences in vocalization• Draw conclusions from data and voila!

• But in reality…

• Put recorders on the monkeys• Gavage with water or alcohol (alternating)• Measure BECs in one hour• Take off recorders• Analyze data for various features• Identify differences in vocalization• Draw conclusions from data and voila!

• But in reality…

Page 19: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 19

ProblemsProblems

1. Exceeding recorder threshold

2. Not enough vocalizations

1. Exceeding recorder threshold

2. Not enough vocalizations

1. Attenuate with rubber and foam

2. Switch to more vocal monkey

1. Attenuate with rubber and foam

2. Switch to more vocal monkey

SolutionsSolutions

Page 20: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 20

Clementine Example WaveformClementine Example Waveform

Page 21: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 21

Data Analysis?Data Analysis?

• Recordings had vocalizations, noise, silence, other monkeys, etc.

• How would we isolate the monkey of interest?

• Recordings had vocalizations, noise, silence, other monkeys, etc.

• How would we isolate the monkey of interest?

Page 22: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 22

Sample SpectrumSample Spectrum

NoiseNoise

VocalizationsVocalizations

Page 23: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 23

Clementine Example SpectrumClementine Example Spectrum

Page 24: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 24

Data AnalysisData Analysis

1. Cut the wave file into smaller segments2. Isolate vocalization parts of speech3. Extract features for vocalization regions4. Compare features for intoxicated versus sober

speech

1. Cut the wave file into smaller segments2. Isolate vocalization parts of speech3. Extract features for vocalization regions4. Compare features for intoxicated versus sober

speech

Page 25: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 25

Segmentation/ClusteringRobust Speaker Change Detection by J. Ajmera et al. (2003)

Segmentation/ClusteringRobust Speaker Change Detection by J. Ajmera et al. (2003)

• Originally created for separating speakers in news broadcasts

• Find likely change points• Segment data with overlapping frames• Cluster similar segments (by speaker)

• Originally created for separating speakers in news broadcasts

• Find likely change points• Segment data with overlapping frames• Cluster similar segments (by speaker)

Page 26: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 26

Segmentation and ClusteringSegmentation and Clustering

Page 27: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 27

Data AnalysisData Analysis

1. Cut the wave file into smaller segments2. Isolate vocalization parts of speech3. Extract features for vocalization regions4. Compare features for intoxicated versus sober

speech

1. Cut the wave file into smaller segments2. Isolate vocalization parts of speech3. Extract features for vocalization regions4. Compare features for intoxicated versus sober

speech

Page 28: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 28

Spectrum – Human vs. MonkeySpectrum – Human vs. Monkey

Page 29: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 29

Results – Human vs. MonkeyResults – Human vs. Monkey

• Bandwidth of formants in monkey vocalizations is larger than for humans

• Humans have more formants (5+), monkeys have much fewer (2-4)

• Distance between the formants for monkeys is much larger than between human formants

• Shape of formants is curved for screams and straight for coos

• Bandwidth of formants in monkey vocalizations is larger than for humans

• Humans have more formants (5+), monkeys have much fewer (2-4)

• Distance between the formants for monkeys is much larger than between human formants

• Shape of formants is curved for screams and straight for coos

Page 30: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 30

Spectrum – Human vs. MonkeySpectrum – Human vs. Monkey

Human Noise Coo Scream

Page 31: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 31

Results - F0 graphsResults - F0 graphs

Page 32: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 32

Results – Alcohol vs. WaterResults – Alcohol vs. Water

• Graphed all of the features

• F0 as x-variable produced most significant results

• F0 tends to be higher during intoxication

• Graphed all of the features

• F0 as x-variable produced most significant results

• F0 tends to be higher during intoxication

Page 33: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 33

Results – Rms vs. f0Results – Rms vs. f0

Page 34: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 34

ResultsResults

• Root mean square (energy) vs. fundamental frequency

• Control vocalizations have larger variation in energy

• Intoxication has higher f0

• Root mean square (energy) vs. fundamental frequency

• Control vocalizations have larger variation in energy

• Intoxication has higher f0

Page 35: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 35

Results – Rms vs. Spec entropyResults – Rms vs. Spec entropy

Page 36: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 36

ResultsResults

• Spectral entropy vs. f0

• Control vocalizations have larger variation in spectral entropy

• Intoxication has higher f0

• Spectral entropy vs. f0

• Control vocalizations have larger variation in spectral entropy

• Intoxication has higher f0

Page 37: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 37

Results Results

• Alcohol increases fundamental frequency (agrees with Hollien study)

• Alcohol decreases variation in energy and spectral entropy

• Consistent with alcohol impairing muscle control of vocal cords

• Alcohol increases fundamental frequency (agrees with Hollien study)

• Alcohol decreases variation in energy and spectral entropy

• Consistent with alcohol impairing muscle control of vocal cords

Page 38: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 38

LimitationsLimitations

• Very small sample size• Limited number of vocalizations• Lots of silence and noise in

recordings• BEC was low (between .017

and .044)• Monkeys were separated – may

have different results in social setting

• Only paired comparisons

• Very small sample size• Limited number of vocalizations• Lots of silence and noise in

recordings• BEC was low (between .017

and .044)• Monkeys were separated – may

have different results in social setting

• Only paired comparisons

Page 39: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 39

In the FutureIn the Future

• Further study correlations between different vocalization features and intoxication

• Use recordings to correlate with other factors (such as stress, dominance, etc.)

• Find ways to increase vocalizations• Pair vocal recordings with visual tracking • Measure ethanol intake using vocalizations in

social settings• Expand studies to other species

• Further study correlations between different vocalization features and intoxication

• Use recordings to correlate with other factors (such as stress, dominance, etc.)

• Find ways to increase vocalizations• Pair vocal recordings with visual tracking • Measure ethanol intake using vocalizations in

social settings• Expand studies to other species

Page 40: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 40

ConclusionConclusion

• Added to the studies done on macaque vocalizations

• Used computer algorithms to separate and analyze data

• Found that formants are a good way to separate human and monkey vocalizations

• Alcohol increases f0 and decreases variability of energy and spectral entropy

• Eventually use vocalizations to measure intoxication in macaques in social settings

• Added to the studies done on macaque vocalizations

• Used computer algorithms to separate and analyze data

• Found that formants are a good way to separate human and monkey vocalizations

• Alcohol increases f0 and decreases variability of energy and spectral entropy

• Eventually use vocalizations to measure intoxication in macaques in social settings

Page 41: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 41

AcknowledgmentsAcknowledgments

• Dr. Kathy Grant• Dr. Izhak Shafran• The Grant Lab (Kevin Nusser, Andrew Rau,

Jessica Shaw, and Cara Candell)• Meysam Asgari• OGI and ONPRC staff and coworkers

• Dr. Kathy Grant• Dr. Izhak Shafran• The Grant Lab (Kevin Nusser, Andrew Rau,

Jessica Shaw, and Cara Candell)• Meysam Asgari• OGI and ONPRC staff and coworkers

Page 42: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 42

Questions?Questions?

Page 43: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 43

Prior Studies - KlingholzPrior Studies - Klingholz

• Approach recognition of intoxication as speaker identification task

• 11 human test subjects and 5 controls• Read a text segment in German• Measure laryngeal and articulatory features

– Laryngeal - fundamental frequency and signal-to-noise (SNR)

– Articulatory – formants (F1/F2 ratio)• Intoxication results

– Increased FO variation– Decreased SNR– Did not change F1/F2

• Correlation between BAL and F0• Long-term voice effort has similar effect• Much more accurate than human recognition

• Approach recognition of intoxication as speaker identification task

• 11 human test subjects and 5 controls• Read a text segment in German• Measure laryngeal and articulatory features

– Laryngeal - fundamental frequency and signal-to-noise (SNR)

– Articulatory – formants (F1/F2 ratio)• Intoxication results

– Increased FO variation– Decreased SNR– Did not change F1/F2

• Correlation between BAL and F0• Long-term voice effort has similar effect• Much more accurate than human recognition

Page 44: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 44

Prior Studies - HollienPrior Studies - Hollien

• Speech samples at four levels of intoxication• 35 human subjects• Results

– Nonfluency increase is best measure– F0 increases and utterance duration increases

(moderate measure)– F0 variability increases (poor measure)– Vocal intensity had no change

• 20% of subjects exhibited no consistent changes

• Speech samples at four levels of intoxication• 35 human subjects• Results

– Nonfluency increase is best measure– F0 increases and utterance duration increases

(moderate measure)– F0 variability increases (poor measure)– Vocal intensity had no change

• 20% of subjects exhibited no consistent changes

Page 45: Monkey See, Monkey Do, Monkey… Talk? by Helen Zou July 23, 2010

Page 45

Prior Studies - WeertsPrior Studies - Weerts

• 33 squirrel monkeys in two different social situations

• Social separation: EtOH reduced isolation peeps• Aggression: EtOH increased aggression peeps• Social context determines effect of drugs

• 33 squirrel monkeys in two different social situations

• Social separation: EtOH reduced isolation peeps• Aggression: EtOH increased aggression peeps• Social context determines effect of drugs