acousteen herman j.m. steeneken subjective intelligibility assessment dr. herman j.m. steeneken
TRANSCRIPT
![Page 1: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/1.jpg)
Acousteen Herman J.M. Steeneken
Subjective Intelligibility Assessment
Dr. Herman J.M. Steeneken
![Page 2: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/2.jpg)
2Acousteen Herman J.M. Steeneken
Signal-to-Noise ratio !!!
![Page 3: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/3.jpg)
3Acousteen Herman J.M. Steeneken
Research Questions
• Intelligibility versus Quality assessment
• Evaluation of a system or application
• Ranking of the performance of a number of systems
• Diagnostic assessment
• Prediction of system performance during design
![Page 4: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/4.jpg)
4Acousteen Herman J.M. Steeneken
Assessment Methods
• Subjective assessment with subjects (speakers
and listeners): representative, limited
reproduction, non diagnostic, laborious
• Objective assessment based on physical
properties (measurements): reproducible,
diagnostic, fast
• Prediction of system performance: design tool
![Page 5: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/5.jpg)
5Acousteen Herman J.M. Steeneken
SUBJECTIVE INTELLIGIBILITY
![Page 6: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/6.jpg)
6Acousteen Herman J.M. Steeneken
Subjective Intelligibility methods
• Phoneme level (nonsense words, rhyme words,
consonants, vowels)
• Word level (meaningful words, nonsense words,
phonetically balanced PB, equally balanced Eqb)
• Sentence level (Mean Opinion Score MOS, Speech
Reception Threshold SRT)
![Page 7: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/7.jpg)
7Acousteen Herman J.M. Steeneken
Methodology I
Response categories:
• Open response (e.g., nonsense words)
• Closed response (Rhyme tests, e.g., MRT, DRT)
• Scaling (MOS, five point scale: excellent - bad)
• Ranking (e.g., pair-wise comparison)
![Page 8: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/8.jpg)
8Acousteen Herman J.M. Steeneken
Methodology II
Test design:
• Words embedded in carrier phrase
• Reference conditions (e.g. MNRU, …)
• Speakers (gender, number, non-native, …)
• Listeners ( number of speaker-listener pairs)
• Learning effects
![Page 9: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/9.jpg)
9Acousteen Herman J.M. Steeneken
Listening test with four subjects
![Page 10: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/10.jpg)
10Acousteen Herman J.M. Steeneken
![Page 11: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/11.jpg)
11Acousteen Herman J.M. Steeneken
Embedded CVC words:
versta des overen nu fijs uithet woord zek eindenoteer lal punt
“Semi random”combinationof:
17 initial consonants15 vowels11 final consonants
![Page 12: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/12.jpg)
12Acousteen Herman J.M. Steeneken
Methodology III
Scoring, data analysis:
• Phone-word scores
• Confusion matrices
• Effective gain (e.g. effective SNR)
• Statistics (Anova, scaling, multiple regression, ...)
![Page 13: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/13.jpg)
13Acousteen Herman J.M. Steeneken
Relation Consonants-Vowels
initial-consonant score (%)
vow
el s
core
(%
)
0
20
40
60
80
100
0 20 40 60 80 100
![Page 14: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/14.jpg)
14Acousteen Herman J.M. Steeneken
How to calculate average word scores
Subject responsesmay require to usethe median
![Page 15: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/15.jpg)
15Acousteen Herman J.M. Steeneken
Example relation MOS-CVC
12345
0 10 20 30 40 50 60 70 80 90 100
mean CVC word score (%)
MOS
![Page 16: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/16.jpg)
16Acousteen Herman J.M. Steeneken
Relation between methods and Qualification
111
Inte
lligi
bilit
y sc
ore
(%
)
0
20
40
60
80
100
0.0 0.2 0.4 0.6 0.8 1.0
PB-words
CVC EQB
STI rbad goodfairpoor excellent
sentences(non-optimized SRT)
![Page 17: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/17.jpg)
17Acousteen Herman J.M. Steeneken
Test-retest variability
Cronbach αbased on split ofspeaker- listenerpairs
![Page 18: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/18.jpg)
18Acousteen Herman J.M. Steeneken
Common Intelligibility scale (IEC60849)
After Barnett and Knight 1994
CIS not linear with SNR
= STI
= 100 - ALcons
x = AI
= PB words (256 words)
= Short Sentences
= PB words (1000 words)
= 1000 syllables
Barnett and Knight (1995)
![Page 19: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/19.jpg)
19Acousteen Herman J.M. Steeneken
CVC scores (%) of realistic conditions
male female
Wide band 90.3 89.3
Telephone band 89.5 85.3
White noise SNR 0 dB 58.0 44.1
Speech noise SNR +3 dB 71.3 60.7
Speech noise SNR -3 dB 43.0 40.6
![Page 20: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/20.jpg)
20Acousteen Herman J.M. Steeneken
Example of consonant confusions
p b f v m n R w
p 1068 62 12 4 4 0 0 2
b 112 1002 0 0 11 7 0 50
f 44 1 915 193 0 0 0 0
v 6 4 337 739 0 0 2 43
m 1 5 0 0 1068 113 1 6
n 0 0 0 0 111 1081 0 2
R 1 2 0 2 0 2 1161 3
w 6 3 1 13 30 7 25 1065
![Page 21: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/21.jpg)
21Acousteen Herman J.M. Steeneken
Two dimensional display of confusions
![Page 22: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/22.jpg)
22Acousteen Herman J.M. Steeneken
Introduction of phoneme specific frequency weighting
Four groups of phonemes (SAMPA notation:
• Fricatives (f, s, v, z)
• Plosives (b, d, x, p, t, k)
• Vowel-like consonants (m, n, l, R, j, w, …)
• Vowels (aa, a, ee, e, o, oo, u, uu, au, …)
![Page 23: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/23.jpg)
23Acousteen Herman J.M. Steeneken
Phoneme group specific spectra
octave-band centre frequency (Hz)
rela
tive
octa
ve-b
and
leve
l (dB
)
0
10
20
30
40
50
125 250 500 1000 2000 4000 A
fricativesplosivesvow-like consvowelsmean (PB)
male speech
![Page 24: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/24.jpg)
24Acousteen Herman J.M. Steeneken
Phoneme group specific spectra
octave-band centre frequency (Hz)
rela
tive
oct
ave
-ba
nd
leve
l (d
B)
0
10
20
30
40
50
125 250 500 1000 2000 4000 A
fricativesplosivesvow-like consvowelsmean (PB)
female speech
![Page 25: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/25.jpg)
25Acousteen Herman J.M. Steeneken
Frequency weighting (fricatives)
octave-band centre frequency (Hz)
freq
uenc
y-w
eigh
ting
fact
or
0.0
0.1
0.2
0.3
0.4
0.5
125 250 500 1k 2k 4k 8k
fricatives male speech female speech
![Page 26: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/26.jpg)
26Acousteen Herman J.M. Steeneken
Frequency weighting (plosives)
octave-band centre frequency (Hz)
fre
qu
en
cy-w
eig
htin
g fa
cto
r
0.0
0.1
0.2
0.3
0.4
0.5
125 250 500 1k 2k 4k 8k
plosives male speech female speech
![Page 27: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/27.jpg)
27Acousteen Herman J.M. Steeneken
Frequency weighting (vowel-like cons)
octave-band centre frequency (Hz)
freq
uenc
y-w
eigh
ting
fact
or
0.0
0.1
0.2
0.3
0.4
0.5
125 250 500 1k 2k 4k 8k
vowel-like consonants male speechfemale speech
![Page 28: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/28.jpg)
28Acousteen Herman J.M. Steeneken
Frequency weightings (vowels)
octave-band centre frequency (Hz)
fre
qu
en
cy-w
eig
htin
g f
act
or
0.0
0.1
0.2
0.3
0.4
0.5
125 250 500 1k 2k 4k 8k
vowels male speech female speech
![Page 29: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/29.jpg)
29Acousteen Herman J.M. Steeneken
Frequency weightings (CVC words)
octave-band centre frequency (Hz)
fre
qu
en
cy-w
eig
htin
g fa
cto
r
0.0
0.1
0.2
0.3
0.4
0.5
125 250 500 1k 2k 4k 8k
CVC words male speech female speech
![Page 30: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/30.jpg)
30Acousteen Herman J.M. Steeneken
Prediction of (CVC) word score by aweighted combination of phoneme group probabilities (DUTCH)
Ci = 0.294 fric + 0.294 plo + 0.412 Cvo
V = V (no weighting)
Cf = 0.273 fric + 0.273 plo + 0.454 Cvo
CVC score = Ci * V * Cf * 100 %
![Page 31: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/31.jpg)
31Acousteen Herman J.M. Steeneken
CVC-word prediction (male)
CVC-word score predicted (%)
CV
C-w
ord
scor
e su
bjec
tive
(%)
0
20
40
60
80
100
0 20 40 60 80 100
male speechS.d.= 4.11%
Male speech
![Page 32: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/32.jpg)
32Acousteen Herman J.M. Steeneken
CVC-word prediction (female)
CVC-word score predicted (%)
CV
C-w
ord
sco
re s
ub
ject
ive
(%
)
0
20
40
60
80
100
0 20 40 60 80 100
female speechS.d. = 3.63%
Female speech
![Page 33: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/33.jpg)
33Acousteen Herman J.M. Steeneken
ISO: Ergonomics – Assessment of speech communication (ISO 9921 DIS)
Application Minimum intelligibilityrating
Maximum vocal effort
Alert and warning situations (correctunderstanding of simple sentences)
Poor Loud see 5.2
Alert and warning situations (correctunderstanding of critical words)
Fair Loud see 5.2
Person-to-person communications(critical)
Fair Loud see 5.3
Person-to-person communications(prolonged normal communication)
Good Normal see 5.3
Public address in public areas Fair Normal see 5.4
Personal communication systems Fair Normal see 5.5
![Page 34: Acousteen Herman J.M. Steeneken Subjective Intelligibility Assessment Dr. Herman J.M. Steeneken](https://reader033.vdocuments.net/reader033/viewer/2022052603/56649c7b5503460f9492ee7e/html5/thumbnails/34.jpg)
34Acousteen Herman J.M. Steeneken
Qualification table
Intelligibilityrating
Sentencescore % Meaningful
PB-wordscore %
CVCEQB-non-sense word
Score %
STISIL
DB
SII
Excellent 100 > 98 > 81 > 0,75 21
Good 100 93 - 98 70 - 81 0,60 - 0,75 15 – 21 > 0,75
Fair 100 80 - 93 53 - 70 0,45 - 0,60 10 – 15
Poor 70 - 100 60 - 80 31 - 53 0,30 - 0,45 3 – 10 < 0,45
Bad < 70 < 60 < 31 < 0,30 < 3