cs 6998 computational approach to emotional speech instructor: prof. julia hirschberg
DESCRIPTION
Julia’s Little Helper : A Real-time Demo of Cantonese/Mandarin Emotional Speech Detection. Suzanne Yuen Mechanical Engineering. William Y. Wang Computer Science. CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg Columbia University 12/21/2009. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/1.jpg)
CS 6998 Computational Approach to Emotional Speech
Instructor: Prof. Julia HirschbergColumbia University 12/21/2009
Julia’s Little Helper:
A Real-time Demo of Cantonese/Mandarin
Emotional Speech Detection
William Y. WangComputer Science
Suzanne YuenMechanical Engineering
![Page 2: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/2.jpg)
Review
1. Target Languages: Cantonese (9 tones) , Mandarin (4 tones)
2. Target Emotions: Anger and Gladness
3. Lexical Features: ASR using a HMM acoustic model trained on
Mandarin Broadcast News [1] and a simple hand-written decoding
dictionary.
4. Prosodic Features: Energy and Tonal Features
5. Real-time drawing of pitch contour, waveform and energy.
6. A text-to-speech agent to greet and teach user how to use this
demo.
[1] Yang Shao, Lan Wang, E-Seminar: an Audio-guide e-Learning System, IEEE International Workshop
on Education Technology and Training (ETT) 2008.
![Page 3: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/3.jpg)
Lexical Scoring 1-3pts Energy 1 pt Tone 1 pt
![Page 4: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/4.jpg)
Dictionary of Affects in Language
by Dr. Cynthia Whissell
Words Pleasantness
Activeness
Imagery
affect 1.7500 1.8571 1.6
affection
2.7778 2.2500 2.0
success 2.8571 1.8000 1.4
successes
3.0000 2.0000 1.4Total words: 8742 words were included. Source: It was actually developed using various sources, for example, college student essays, interviews and teenagers description of their own emotion state. So, it can have a broad coverage and avoid biased data.
![Page 5: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/5.jpg)
Sentence Lexical Scoring
Words Pleasantness
I 2.3750
won 2.5556
Best 2.5455
Paper 1.2857
Award 2.8333
“I won best paper award!”
Score = (2.375 + 2.5556 + 2.5455 + 1.2857 + 2.8333) / 5 = 2.319
![Page 6: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/6.jpg)
Machine Translation
English Word
Pleasantness Chinese Translation
alcohol 1.7143 酒精 (alcohol)
alcoholic 1.0000 酒精 (alcohol)
muscles 2.3333 肌肉 (muscles)
muscular 2.6250 肌肉 (muscles)
Multilingual Challenges: English Chinese
![Page 7: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/7.jpg)
Encoding and Mapping
Mandarin Word
Pinyin(Phone Set)
Corresponding Cantonese
PinyinMapping
喜欢 (like) xi3 hua1n 喜欢 (like) hei2 fu1n
睡觉 (sleep) shui4 jiao4 训觉 (sleep) fen4 gao4
Tasks:1. Mandarin Pinyin (Phone set used by Acoustic Model)2. Mandarin Cantonese
Note that not all words in Mandarin have theirs’ exact and direct mappings in Cantonese words and vice versa.
3. Cantonese Pinyin
![Page 8: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/8.jpg)
Text-to-speech Engine
1. Implement the text-to-speech engine.
2. “Play with” a text-to-speech engine.
3. Engine: TruVoice
Lernout & Hauspie Speech Products, or L&H
Went bankrupt in 2001
technology now owned by Nuance
![Page 9: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/9.jpg)
L&H TTS Functionality
• Developed in 1997
• Advanced text pre-processing and no vocabulary restrictions
• User-definable pronunciation dictionary
• Accurately pronounces surnames and place names
• Flexible pitch, volume and speech rate
• Intonation support for punctuation
![Page 10: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/10.jpg)
Test Overview
1. Participants –
1. gender: 6 male, 6 female
2. Native Language – 6 Mandarin, 6 Cantonese
2. Two Parts
1. JLH module and self-rating (24 lines total)
2. Perception test – Rating lines from others (72)
![Page 11: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/11.jpg)
Sentences
1. Three types – questions, exclamations, statements
2. Randomized order of sentences for each participant
3. Examples:
In Lexicon Not In Lexicon
Angry & Not Angry
Insane! 痴 (chi1) 线 (xian4) !
You're wrong. 你 (ni3) 错 (cuo4) 了(le,liao3,liao4) 。
Glad & Not Glad When's the wedding? 什 (shen2,shi2) 么(ma,me) 时 (shi2) 候(hou4) 结 (jie1,jie2) 婚 (hun1) ?
It's bubbly? 有 (you3,you4) 泡(pao1,pao4) 泡(pao1,pao4) 吗(ma2,ma3,ma) ?
![Page 12: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/12.jpg)
Analysis
1. Plan to examine differences and affects of following:
1. Ratings - JLH star rating, self rating, & 3 perception ratings
2. Language – Cantonese, Mandarin
3. Gender – female, male
4. Sentence structure – exclamation, question, & statement
2. Interesting points –
1. Huge range of Chinese accents
2. Tones of words may change depending on previous words
(such as English a mug vs. an umbrella)
3. Variations in colloquial speech, addressed by using Chinese
script
![Page 13: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/13.jpg)
Future Work
1. Improve the prosodic analysis. More features should be explored.
2. Improve the lexical scoring. Use POS tagger or other NLP tools to
weigh different constituents of recognized sentence.
3. Finer-grain the emotion types and investigate the differences.
4. Study translational divergence in English-Chinese MT .
![Page 14: CS 6998 Computational Approach to Emotional Speech Instructor: Prof. Julia Hirschberg](https://reader034.vdocuments.net/reader034/viewer/2022051622/56815086550346895dbe82fe/html5/thumbnails/14.jpg)
Demo