the art and science of grammar coverage analysis

The Art and Science ofGrammar Coverage Analysis

Dominique BoucherNu Echo Inc.dominique.boucher@nuecho.com

SpeechTEK 2009New York, USA

The grammar development processEnter / get initial set of sentences in coverage set

Write initial grammar to cover

utterances

Run coverage tests

Fix grammaryesProblems?no

Enrich coverage set

Generate sentences

Objective: To obtain:(a) A complete coverage set (b) A grammar that covers the coverage set

and produces the correct semantic result

The importance of coverage analysis

• Ensure grammars conform to their specification• Semantic tags testing• Ensure grammars conform to their specification• Semantic tags testing

Design and development

• Grammars evolve over the life of an application• Provides an effective tool for testing that a grammar

isn’t accidentally broken by a grammar change

• Grammars evolve over the life of an application• Provides an effective tool for testing that a grammar

isn’t accidentally broken by a grammar change

Maintenance and optimization

• Ensure proper conversion between grammar formats

• Ensure proper conversion between grammar formats

Conversion projects

Grammar coverage challenges

Provide exhaustive coverage of all sentence patterns… … with the smallest possible set of sentences.

Otherwise:– Analysis of generated sentences will be time-consuming; and– Errors will go undetected

Two complementary techniques

#1 Sentence

Generation

#2 Sentence patterns

Exploration

Technique #1Sentence generation

Tools usually provided by the ASR engine SDK– Operate on the source or compiled grammar

Commonly used generation strategies– Exhaustive generation– Generation of a fixed number of random sentences

Technical difficulties

Some grammars generate an infinite number of sentences– Exhaustive generation not possible

Semantic tags not all tested– Errors may remain undetected until application run time

All interesting cases may not be covered by the generated sentences

Uninteresting patterns are generated over and over

Sentence generation best practices

Avoid generating redundant sentences– Powerful sentence generation tools makes this possible– Too many sentences increases the risk of errors going

undetected

Carefully examine generated sentences– If a sentence doesn’t look right, it probably shouldn’t be in

there (although, to be sure, look at the parse tree)

Make sure the coverage test is as complete as possible– Should include all semantic tags and all sentence patterns– Full coverage is best (whenever possible)

Sentence generation revisited

Individual rule configuration

More effective strategies – Tags coverage– All grammar paths– Pick from @examples– Use fixed sentence

Generation can be started from any set of sentence patterns

Sentence generation tool

Interactive Demo

Technique #2Exploring sentence patterns

Interactive expansion of grammar rules

Derive sentence patterns– Useful to generate

sentences for a specific pattern

Derive complete sentences– Can be debugged, etc.

Ideal to understand the structure of a grammar

Sentence explorer tool

Interactive Demo

Three typical problems…… and how to fix them!

Interactive Demo