Rapid and Accurate Rapid and Accurate Spoken Term DetectionSpoken Term Detection

Owen Kimball

BBN Technologies

15 December 2006

15-Dec-06Rapid and Accurate Spoken Term Detection 2

Overview of TalkOverview of Talk

• BBN Levantine system description

• Evaluation results

• Diacritics

• Out-of-vocabulary issues

15-Dec-06Rapid and Accurate Spoken Term Detection 3

BBN Evaluation TeamBBN Evaluation Team

Core Team• Chia-lin Kao• Owen Kimball• Michael Kleber• David Miller

Additional assistance• Thomas Colthurst• Herb Gish• Steve Lowe• Rich Schwartz

15-Dec-06Rapid and Accurate Spoken Term Detection 4

BBN System OverviewBBN System Overview

Byblos STT

indexer

detector

decider

latticesphonetic-transcripts

indexscored

detectionlists

final outputwith YES/NO

decisions

audiosearc

hterms

ATWV cost

parameters

15-Dec-06Rapid and Accurate Spoken Term Detection 5

Levantine STT ConfigurationLevantine STT Configuration

• STT generates a lattice of hypotheses and a phonetic transcript for each input file.

• Word-based system:– Orthography based on Modern Standard Arabic

(MSA), no short vowel diacritics– Acoustic: 57.3 hours LDC

(noise words, no mixture exponents)– Language: 250 hours of data, 1.3M words

• 38.5K dictionary, grapheme-as-phoneme based plus 100 manual pronunciations

– unknown short vowel (U), 39 phonemes

• 42.32% WER on STD Dev06 CTS data

15-Dec-06Rapid and Accurate Spoken Term Detection 6

Levantine CTS ResultsLevantine CTS Results

0.3467Eval06

0.410DryRun

0.515Dev06

ATWV Data

15-Dec-06Rapid and Accurate Spoken Term Detection 7

OOV Pipeline: DetectorOOV Pipeline: Detector

• Word-based STT produces 1-best transcript: pronounce it 1-best phonetic transcript.

• Query is OOV if it contains any OOV word.

• OOV query detection:– Pronounce query (grapheme-as-phoneme)– Find minimal edit-distance alignments (agrep)– Score = % error = phonemes#

distanceedit 1

15-Dec-06Rapid and Accurate Spoken Term Detection 8

OOV Pipeline: DeciderOOV Pipeline: Decider

• Need different Yes/No decision procedure:IV-decider requires posterior probabilities.

• Simple OOV decision procedure:– Constant threshold on score (~ 0.7)– Cap on maximum number of hits (0-3)– Values set to maximize ATWV on Dev06 data.

15-Dec-06Rapid and Accurate Spoken Term Detection 9

OOV Pipeline: ResultsOOV Pipeline: Results

• ATWV remained good:0.3450 IV

0.3635 OOV

• Searches take longer: ~10-15x IV speed on Dev06 and DryRun06,

with no attempt at indexing.

15-Dec-06Rapid and Accurate Spoken Term Detection 10

OOV Directions for ImprovementOOV Directions for Improvement

• Score substitutions using phoneme confusion matrix instead of flat edit distance

• Speed: indexing phonetic transcripts for approximate matching

• Search lattices beyond 1-best transcripts

15-Dec-06Rapid and Accurate Spoken Term Detection 11

Levantine Diacritic IssuesLevantine Diacritic Issues

• Originally looked at diacritized Levantine

• Trained STT engine using LDC 45 hour set

• Ran STD without knowing WER (no diacritized STT test set to measure WER).– Found very high false alarm rate

• Examining FAs found hits that were legitimate alternate spellings

15-Dec-06Rapid and Accurate Spoken Term Detection 12

Levantine Diacritics- Alternate SpellingsLevantine Diacritics- Alternate Spellings

• Examining query words found more of same:– In first 22 terms of dry run term list, 14 are “alternate

diacritic” spellings of 5 underlying words, i.e. there were just 13 unique words in the first 22 terms

– Min~ahumo v Minohumo

– AlHayaApi v AlHayaAp

– Waliko v Walika

– qabilo v qabola v qabolo

• LDC training and STD test set had additional pervasive differences

15-Dec-06Rapid and Accurate Spoken Term Detection 13

No-Diacritic Levantine IssuesNo-Diacritic Levantine Issues

• A quick look turned up a smaller number of problems for no-diacritic Levantine– Looking at 7 top-FA terms in dev set, found

• “bHky” vs “b>Hky” but no other spelling confusions

• One ref instance of term with 0 duration

• It would be interesting to QC test sets for inconsistent spellings and other issues