Spatial reasoning in a multi-modal user guide for a complex machine

Nadejda Soudzilovskaia, Rafael Bidarra, Frederik W. Jansen

Delft University of Technology,Faculty of Information Technology and Systems

Contents

Problem introduction Application Geometry based reasoning Techniques Examples Questions

Problem introduction Multimodal interaction

– Multimodal = speech + graphics + text• Computer speech needs feedback of text and graphics• Speech introduces geometric references,

(e.g.the object in front of the building)

• Solving of these references needs combination of geometric and functional reasoning

?

Application: Interactive fax user guide Speech & Reasoning & Graphics Task: develop dialogue technologies

Application

Single questions & answers about the fax Questions: Speech Answers: Graphics & Speech

How can I make a copy of a document?

• Put the document into the paper source tray

• Press copy button

Fax encyclopedia

Pre-programmed answers– develop set of possible user questions– develop answer + “movie” for every question

Possible solutions

Not an intelligent system. Only pre-cooked question and answers are possible

Possible solutions

Answers derived by reasoning

– knowledge based system containing:

• data about fax functionality

• data about fax construction (3D model)

– Reasoning system based on:

• semantic

• geometry

Answers derived by reasoning

– knowledge based system containing:

• data about fax functionality

• data about fax construction (3D model)

– Reasoning system based on:

• semantics

• geometry

System architecture

Dialogue manager

Speech recognizer

Parser

Geometrybased

reasoning

Semanticbased

reasoning

Graphical answer

generation

Textual answer

generation

?

Graphics

– Solve geometric references

– Derive descriptions of a component location

– Show components

Tasks:

Geometry based reasoning

Tasks:

– Navigation though the virtual world

How can I make a copy of a document?

• Put the document into the paper source tray

• Press copy button

Geometry based reasoning Show components

Geometry based reasoning

– choose appropriate scale– choose appropriate view point

Show components

Where is the power slot?

Power slot is on theback side of the fax

Geometry based reasoning Deriving location descriptions

What is the button left of the start button?

This is a copy button

Geometry based reasoning Solve geometric references

Data about fax construction

– hierarchy graph

– feature graph

– geometry graph

Geometry based reasoning

Top face

Rib

Bottom face

CornerCorner

vertexedgeedge

face face

Telefax

Corpus Telephone Powerplug

Telephonereceiver

Operational panel

LCD screen

Start button

Keypad

....

....

Problem: search for a suitable reference object Proposal: use of hierarchy Result: reduction of reasoning complexity

Techniques

Telefax

Corpus Telephone Powerplug

Telephonereceiver

Operational panel

LCD screen

Start button

Keypad

....

....

Analyze of dimensionality in view of children location

Techniques

Objects that do not have children nodes:• Leaf objects

Objects that have children nodes:

• 3D objects: fax corpus, telephone

• 2D objects: operational panel, slot panel

• 1D objects: cables

• Set objects: group of slots, group of switches

Use of applicability zone in for a geometric reference

• when reason about a position on a surface of another object the spatial reference differ for 3D object and 2D object

• use of fuzzy logic to process zones of applicability

Techniques

Where is the power slot?

Power slot is on theback side of the fax

Examples Deriving location descriptions

1 priority Location on the surface

1.1 on the corner 1.2 on the right / left side 1.3 on the top / bottom face 1.4 on the front / back face

• Search for the object parent

• Select meaningful descriptions

2 priority Location relatively to neighbor objects

2.1 between (two other objects) 2.2 in front of , under, above 2.3 to the left of 2.4 to the left of 2.5 behind, under 2.6. near

Developed answer : The printer slot is on the back face of the fax, to the left of the printer slot

Deriving location descriptions

What is the button left of the start button?

This is a copy button

Examples Solve geometric references

1. Search through the same hierarchy level

1.1 Search for a parent group of operational buttons

1.2 Analyze location left of the start button

1.3. Choose suitable objects

2. Go one level higher in the hierarchy

2.1 Search for a parent of the parentoperational panel

2.2 Analyze location left of the start button

2.3 Select objects

3. Go one level higher……...

Solve geometric references

Summary

Role of spatial reasoning techniques in the application of multi-modal user guides

A fax machine user guide system Untilisng 3D model of a fax Use of hierarchy Use of dimentionality Use of areference applicability zone

Conclusions

Geometry based reasoning is a new functionality for such a dialogue system

3D model of the fax-machine is used as a source of data for reasoning

Several techniques for the geometry-based reasoning were developed and implemented using Java3D and OpenGL languages

Questions

?