cs 224s / linguist 281 speech recognition, synthesis, and dialogue dan jurafsky lecture 5: intro to...
TRANSCRIPT
![Page 1: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/1.jpg)
CS 224S / LINGUIST 281Speech Recognition, Synthesis, and
Dialogue
Dan Jurafsky
Lecture 5: Intro to ASR+HMMs:Forward, Viterbi, Word Error Rate
IP Notice:
![Page 2: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/2.jpg)
Outline for Today
• Speech Recognition Architectural Overview• Hidden Markov Models in general and for
speech Forward Viterbi Decoding
• How this fits into the ASR component of course July 27 (today): HMMs, Forward, Viterbi, Jan 29 Baum-Welch (Forward-Backward) Feb 3: Feature Extraction, MFCCs Feb 5: Acoustic Modeling and GMMs Feb 10: N-grams and Language Modeling Feb 24: Search and Advanced Decoding Feb 26: Dealing with Variation Mar 3: Dealing with Disfluencies
![Page 3: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/3.jpg)
LVCSR
• Large Vocabulary Continuous Speech Recognition
• ~20,000-64,000 words• Speaker independent (vs. speaker-
dependent)• Continuous speech (vs isolated-word)
![Page 4: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/4.jpg)
Current error rates
Task Vocabulary Error Rate%
Digits 11 0.5
WSJ read speech 5K 3
WSJ read speech 20K 3
Broadcast news 64,000+ 10
Conversational Telephone
64,000+ 20
Ballpark numbers; exact numbers depend very much on the specific corpus
![Page 5: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/5.jpg)
HSR versus ASR
• Conclusions:Machines about 5 times worse than humansGap increases with noisy speechThese numbers are rough, take with grain of
salt
Task Vocab
ASR Hum SR
Continuous digits
11 .5 .009
WSJ 1995 clean 5K 3 0.9
WSJ 1995 w/noise
5K 9 1.1
SWBD 2004 65K 20 4
![Page 6: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/6.jpg)
LVCSR Design Intuition
• Build a statistical model of the speech-to-words process
• Collect lots and lots of speech, and transcribe all the words.
• Train the model on the labeled speech
• Paradigm: Supervised Machine Learning + Search
![Page 7: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/7.jpg)
The Noisy Channel Model
• Search through space of all possible sentences.
• Pick the one that is most probable given the waveform.
![Page 8: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/8.jpg)
The Noisy Channel Model (II)
• What is the most likely sentence out of all sentences in the language L given some acoustic input O?
• Treat acoustic input O as sequence of individual observations O = o1,o2,o3,…,ot
• Define a sentence as a sequence of words:W = w1,w2,w3,…,wn
![Page 9: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/9.jpg)
Noisy Channel Model (III)
• Probabilistic implication: Pick the highest prob S:
• We can use Bayes rule to rewrite this:
• Since denominator is the same for each candidate sentence W, we can ignore it for the argmax:
€
ˆ W = argmaxW ∈L
P(W | O)
€
ˆ W = argmaxW ∈L
P(O |W )P(W )€
ˆ W = argmaxW ∈L
P(O |W )P(W )
P(O)
![Page 10: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/10.jpg)
Speech Recognition Architecture
![Page 11: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/11.jpg)
Noisy channel model
€
ˆ W = argmaxW ∈L
P(O |W )P(W )
likelihood prior
![Page 12: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/12.jpg)
The noisy channel model
• Ignoring the denominator leaves us with two factors: P(Source) and P(Signal|Source)
![Page 13: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/13.jpg)
Speech Architecture meets Noisy Channel
![Page 14: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/14.jpg)
Architecture: Five easy pieces (only 2 for today)
• Feature extraction• Acoustic Modeling• HMMs, Lexicons, and Pronunciation• Decoding• Language Modeling
![Page 15: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/15.jpg)
Lexicon
• A list of words• Each one with a pronunciation in
terms of phones• We get these from on-line
pronucniation dictionary• CMU dictionary: 127K words
http://www.speech.cs.cmu.edu/cgi-bin/cmudict
• We’ll represent the lexicon as an HMM
15
![Page 16: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/16.jpg)
HMMs for speech
![Page 17: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/17.jpg)
Phones are not homogeneous!
Time (s)0.48152 0.937203
0
5000
ay k
![Page 18: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/18.jpg)
Each phone has 3 subphones
![Page 19: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/19.jpg)
Resulting HMM word model for “six”
![Page 20: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/20.jpg)
HMM for the digit recognition task
04/19/23 20Speech and Language Processing Jurafsky and Martin
![Page 21: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/21.jpg)
More formally: Toward HMMs
• A weighted finite-state automaton An FSA with probabilities onthe arcs The sum of the probabilities leaving any arc
must sum to one
• A Markov chain (or observable Markov Model) a special case of a WFST in which the input
sequence uniquely determines which states the automaton will go through
• Markov chains can’t represent inherently ambiguous problems Useful for assigning probabilities to
unambiguous sequences
![Page 22: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/22.jpg)
Markov chain for weather
![Page 23: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/23.jpg)
Markov chain for words
![Page 24: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/24.jpg)
Markov chain = “First-order observable Markov Model”
• a set of states Q = q1, q2…qN; the state at time t is qt
• Transition probabilities: a set of probabilities A = a01a02…an1…ann.
Each aij represents the probability of transitioning from state i to state j
The set of these is the transition probability matrix A
• Distinguished start and end states€
aij = P(qt = j | qt−1 = i) 1 ≤ i, j ≤ N
€
aij =1; 1 ≤ i ≤ Nj=1
N
∑
![Page 25: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/25.jpg)
Markov chain = “First-order observable Markov Model”
• Current state only depends on previous state
€
Markov Assumption : P(qi | q1...qi−1) = P(qi | qi−1)
![Page 26: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/26.jpg)
Another representation for start state
• Instead of start state
• Special initial probability vector An initial distribution over probability of start states
• Constraints:
€
i = P(q1 = i) 1 ≤ i ≤ N
€
j =1j=1
N
∑
![Page 27: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/27.jpg)
The weather figure using pi
![Page 28: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/28.jpg)
The weather figure: specific example
![Page 29: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/29.jpg)
Markov chain for weather
• What is the probability of 4 consecutive warm days?
• Sequence is warm-warm-warm-warm• I.e., state sequence is 3-3-3-3• P(3,3,3,3) =
3a33a33a33a33 = 0.2 x (0.6)3 = 0.0432
![Page 30: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/30.jpg)
How about?
• Hot hot hot hot• Cold hot cold hot
• What does the difference in these probabilities tell you about the real world weather info encoded in the figure?
![Page 31: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/31.jpg)
HMM for Ice Cream
• You are a climatologist in the year 2799
• Studying global warming• You can’t find any records of the
weather in Baltimore, MD for summer of 2008
• But you find Jason Eisner’s diary• Which lists how many ice-creams
Jason ate every date that summer• Our job: figure out how hot it was
![Page 32: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/32.jpg)
Hidden Markov Model
• For Markov chains, the output symbols are the same as the states. See hot weather: we’re in state hot
• But in named-entity or part-of-speech tagging (and speech recognition and other things) The output symbols are words But the hidden states are something else
Part-of-speech tags Named entity tags
• So we need an extension!• A Hidden Markov Model is an extension of a
Markov chain in which the input symbols are not the same as the states.
• This means we don’t know which state we are in.
![Page 33: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/33.jpg)
Hidden Markov Models
![Page 34: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/34.jpg)
Assumptions
• Markov assumption:
• Output-independence assumption
€
P(qi | q1...qi−1) = P(qi | qi−1)
€
P(ot | O1t−1,q1
t ) = P(ot |qt )
![Page 35: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/35.jpg)
Eisner task
• Given Ice Cream Observation Sequence:
1,2,3,2,2,2,3…
• Produce:Weather Sequence: H,C,H,H,H,C…
![Page 36: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/36.jpg)
HMM for ice cream
![Page 37: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/37.jpg)
Different types of HMM structure
Bakis = left-to-right Ergodic = fully-connected
![Page 38: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/38.jpg)
The Three Basic Problems for HMMs
• Problem 1 (Evaluation): Given the observation sequence O=(o1o2…oT), and an HMM model = (A,B), how do we efficiently compute P(O| ), the probability of the observation sequence, given the model
• Problem 2 (Decoding): Given the observation sequence O=(o1o2…oT), and an HMM model = (A,B), how do we choose a corresponding state sequence Q=(q1q2…qT) that is optimal in some sense (i.e., best explains the observations)
• Problem 3 (Learning): How do we adjust the model parameters = (A,B) to maximize P(O| )?
Jack Ferguson at IDA in the 1960s
![Page 39: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/39.jpg)
Problem 1: computing the observation likelihood
• Given the following HMM:
• How likely is the sequence 3 1 3?
![Page 40: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/40.jpg)
How to compute likelihood
• For a Markov chain, we just follow the states 3 1 3 and multiply the probabilities
• But for an HMM, we don’t know what the states are!
• So let’s start with a simpler situation.• Computing the observation likelihood for
a given hidden state sequence Suppose we knew the weather and wanted to
predict how much ice cream Jason would eat. I.e. P( 3 1 3 | H H C)
![Page 41: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/41.jpg)
Computing likelihood of 3 1 3 given hidden state sequence
![Page 42: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/42.jpg)
Computing joint probability of observation and state sequence
![Page 43: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/43.jpg)
Computing total likelihood of 3 1 3
• We would need to sum over Hot hot cold Hot hot hot Hot cold hot ….
• How many possible hidden state sequences are there for this sequence?
• How about in general for an HMM with N hidden states and a sequence of T observations? NT
• So we can’t just do separate computation for each hidden state sequence.
![Page 44: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/44.jpg)
Instead: the Forward algorithm
• A kind of dynamic programming algorithm Just like Minimum Edit Distance Uses a table to store intermediate values
• Idea: Compute the likelihood of the observation
sequence By summing over all possible hidden state
sequences But doing this efficiently
By folding all the sequences into a single trellis
![Page 45: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/45.jpg)
The forward algorithm
• The goal of the forward algorithm is to compute
• We’ll do this by recursion
€
P(o1,o2,...,oT ,qT = qF | λ )
![Page 46: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/46.jpg)
The forward algorithm
• Each cell of the forward algorithm trellis alphat(j)Represents the probability of being in
state jAfter seeing the first t observationsGiven the automaton
• Each cell thus expresses the following probabilty
![Page 47: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/47.jpg)
The Forward Recursion
![Page 48: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/48.jpg)
The Forward Trellis
![Page 49: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/49.jpg)
We update each cell
![Page 50: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/50.jpg)
The Forward Algorithm
![Page 51: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/51.jpg)
Decoding
• Given an observation sequence 3 1 3
• And an HMM• The task of the decoder
To find the best hidden state sequence
• Given the observation sequence O=(o1o2…oT), and an HMM model = (A,B), how do we choose a corresponding state sequence Q=(q1q2…qT) that is optimal in some sense (i.e., best explains the observations)
![Page 52: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/52.jpg)
Decoding
• One possibility: For each hidden state sequence Q
HHH, HHC, HCH, Compute P(O|Q) Pick the highest one
• Why not?NT
• Instead: The Viterbi algorithm Is again a dynamic programming algorithmUses a similar trellis to the Forward algorithm
![Page 53: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/53.jpg)
Viterbi intuition
• We want to compute the joint probability of the observation sequence together with the best state sequence
€
maxq 0,q1,...,qT
P(q0,q1,...,qT ,o1,o2,...,oT ,qT = qF | λ )
![Page 54: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/54.jpg)
Viterbi Recursion
![Page 55: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/55.jpg)
The Viterbi trellis
![Page 56: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/56.jpg)
Viterbi intuition
• Process observation sequence left to right
• Filling out the trellis• Each cell:
![Page 57: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/57.jpg)
Viterbi Algorithm
![Page 58: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/58.jpg)
Viterbi backtrace
![Page 59: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/59.jpg)
HMMs for Speech
• We haven’t yet shown how to learn the A and B matrices for HMMs; we’ll do that on Thursday The Baum-Welch (Forward-Backward
alg)
• But let’s return to think about speech
![Page 60: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/60.jpg)
Reminder: a word looks like this:
![Page 61: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/61.jpg)
HMM for digit recognition task
![Page 62: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/62.jpg)
The Evaluation (forward) problem for speech
• The observation sequence O is a series of MFCC vectors
• The hidden states W are the phones and words
• For a given phone/word string W, our job is to evaluate P(O|W)
• Intuition: how likely is the input to have been generated by just that word string W
![Page 63: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/63.jpg)
Evaluation for speech: Summing over all different paths!
• f ay ay ay ay v v v v • f f ay ay ay ay v v v • f f f f ay ay ay ay v• f f ay ay ay ay ay ay v• f f ay ay ay ay ay ay ay ay v• f f ay v v v v v v v
![Page 64: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/64.jpg)
The forward lattice for “five”
![Page 65: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/65.jpg)
The forward trellis for “five”
![Page 66: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/66.jpg)
Viterbi trellis for “five”
![Page 67: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/67.jpg)
Viterbi trellis for “five”
![Page 68: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/68.jpg)
Search space with bigrams
![Page 69: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/69.jpg)
Viterbi trellis
69
![Page 70: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/70.jpg)
Viterbi backtrace
70
![Page 71: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/71.jpg)
Evaluation
• How to evaluate the word string output by a speech recognizer?
![Page 72: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/72.jpg)
Word Error Rate
• Word Error Rate = 100 (Insertions+Substitutions + Deletions) ------------------------------ Total Word in Correct TranscriptAligment example:REF: portable **** PHONE UPSTAIRS last night soHYP: portable FORM OF STORES last night soEval I S S WER = 100 (1+2+0)/6 = 50%
![Page 73: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/73.jpg)
NIST sctk-1.3 scoring softare:Computing WER with sclite
• http://www.nist.gov/speech/tools/• Sclite aligns a hypothesized text (HYP) (from the
recognizer) with a correct or reference text (REF) (human transcribed)
id: (2347-b-013)
Scores: (#C #S #D #I) 9 3 1 2
REF: was an engineer SO I i was always with **** **** MEN UM and they
HYP: was an engineer ** AND i was always with THEM THEY ALL THAT and they
Eval: D S I I S S
![Page 74: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/74.jpg)
Sclite output for error analysis
CONFUSION PAIRS Total (972) With >= 1 occurances (972)
1: 6 -> (%hesitation) ==> on 2: 6 -> the ==> that 3: 5 -> but ==> that 4: 4 -> a ==> the 5: 4 -> four ==> for 6: 4 -> in ==> and 7: 4 -> there ==> that 8: 3 -> (%hesitation) ==> and 9: 3 -> (%hesitation) ==> the 10: 3 -> (a-) ==> i 11: 3 -> and ==> i 12: 3 -> and ==> in 13: 3 -> are ==> there 14: 3 -> as ==> is 15: 3 -> have ==> that 16: 3 -> is ==> this
![Page 75: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/75.jpg)
Sclite output for error analysis
17: 3 -> it ==> that 18: 3 -> mouse ==> most 19: 3 -> was ==> is 20: 3 -> was ==> this 21: 3 -> you ==> we 22: 2 -> (%hesitation) ==> it 23: 2 -> (%hesitation) ==> that 24: 2 -> (%hesitation) ==> to 25: 2 -> (%hesitation) ==> yeah 26: 2 -> a ==> all 27: 2 -> a ==> know 28: 2 -> a ==> you 29: 2 -> along ==> well 30: 2 -> and ==> it 31: 2 -> and ==> we 32: 2 -> and ==> you 33: 2 -> are ==> i 34: 2 -> are ==> were
![Page 76: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/76.jpg)
Better metrics than WER?
• WER has been useful• But should we be more concerned with
meaning (“semantic error rate”)? Good idea, but hard to agree on Has been applied in dialogue systems, where
desired semantic output is more clear
![Page 77: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/77.jpg)
Summary: ASR Architecture
• Five easy pieces: ASR Noisy Channel architecture1) Feature Extraction:
39 “MFCC” features
2) Acoustic Model: Gaussians for computing p(o|q)
3) Lexicon/Pronunciation Model• HMM: what phones can follow each other
4) Language Model• N-grams for computing p(wi|wi-1)
5) Decoder• Viterbi algorithm: dynamic programming for
combining all these to get word sequence from speech!
77
![Page 78: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/78.jpg)
ASR Lexicon: Markov Models for pronunciation
![Page 79: CS 224S / LINGUIST 281 Speech Recognition, Synthesis, and Dialogue Dan Jurafsky Lecture 5: Intro to ASR+HMMs: Forward, Viterbi, Word Error Rate IP Notice:](https://reader030.vdocuments.net/reader030/viewer/2022032600/56649dbc5503460f94aae7f5/html5/thumbnails/79.jpg)
Summary
• Speech Recognition Architectural Overview
• Hidden Markov Models in general ForwardViterbi Decoding
• Hidden Markov models for Speech• Evaluation