mobile multilingual maintenance man

1

MobileMultilingualMaintenanceMan


2


4M Assists a Service Person

3


4M supports Problem Solving

Problem situation Possible solution Problem solved

Consult Do it ReportPlan

Service Center

Support for reporting

Speech recognition Dialogue anagement

Message understanding Situational recognition

Search for information Presentation of instructions

Accumulation of knowledge

Creation of basic knowledge

Multilinguality

4


Information Retrieval

INPUT ANALYZER

Model-based Tool

4M Server Interface

Desktop clients(text, speech, graphics)

4M Clients

Mobile clients (menus, speech)

FACTS REASONER

ONTOLOGY SERVICES

Ontologyrepository

OUTPUT GENERATOR

Human Assistance

Case-based Tool

DIALOGUE MANAGER

Speech recognition on either client or

server side

Asynchronic query to tools

Speech synthesis on either client or

server side

1

12

2

113

4 10

9

58

67

Selected tool response

Conceptual system

response

Verbalized system

response

Conceptualized user input

4M Architecture

5


Information RetrievalModel-based

DATA SOURCES

Text/XML files

4Mdata

External systems

Externalsystemsand data

4M System

FACT FINDING & REASONING TOOLS ONTOLOGY SERVICES

Ontologyrepository

OUTPUT GENERATOR

RDF/XML Generator

Report Generator

Human AssistanceCase-based…

DIALOGUE MANAGER

Input Module

Dialogue Context

Response Evaluator

Discourse Memory

Reasoner Interface

Dialogue Planner

Text (documentation, emails etc.)

INFORMATION PROCESSING

Segmentation Annotation

ONTOLOGY PROCESSING

Designer Compiler

INPUT ANALYZER

Parser

Concept Analysis

DIALOGUE RDF RULES

TriplEd

4M Knowledge Accumulation

6


CoGKS assists a Service Community with 4M

7


CoGKS Clients

CoGKS Server

Desktop clients(text, speech, graphics)

Mobile clients (text, speech)

Access manager

Room manager

Room (Job)

ChatSummary Attacheddata

4M System

Summarization of cases

Active or passive assistance mode

Import supporting

material

Establish rooms

Query and copy rooms

Consult participants and solve problems in cooperation

Invite participants

Variety of reasoning: Model, Case, IR,

Human, …Cumulative

knowledge baseConsult 4M

Speech, text

The Cognitive Guidance and Knowledge System (CoGKS)

8


4M Ontology: a multilingual ontology matrix

company terms

domain termsdomain lexicon

generic terms

domain ontology

genericontology

generic lexicon

company termscompany lexicon company ontology

company instance bases

9


4M Dialogue Manager

Constructive Dialogue Management (CDM)Input Analysis

and Natural Language Generation in 4M Multilingual Agenda Markup Language.

Agent Based Dialogue Model Specifying Query Selection Weighted Rules for Output Responses Dialogue Planning Utilizing Dialogue Objects

Reasoner Data Interfaces in Uniform RDF Blackboard Data Architecture. Reasoner Abstraction Broadcast Messages

10


Constructive Dialogue Management theory (CDM)

(Jokinen, 1996)

Human and Machine Participate in Ideal Cooperation (Allwood, 1976)

strive to achieve the same purposes

cognitively and ethically consider each other in trying to achieve these purposes

trust each other to act according to the above principles

4M Dialogue Manager

11


Agent Based Dialogue Model Specifying Query Selection Weighted Rules for Output Responses

DM

Question Pool Sorted by Probability:

0.21: [pwr1] [may_be] [off] 0.08: [inet1] [may_be] [down] 0.04: [hub1] [may_be] [broken]

“Is the power switch off?”

Q1

Q2

Q3

4M Dialogue Manager

12


DM

R1

R2

R3

Collected facts:

[network] [is_not] [accessible] [eth_cable] [is_not] [loose] [pwr1] [may_be] [off] [hub1] [may_be] [broken]

Model Based Reasoning Case Based Reasoning Information Retrieval

4M Dialogue Manager

Reasoner Data Interfaces in Uniform RDF Blackboard Data Architecture – All current data is in messages. Reasoner Abstraction – New reasoners can be added easily. Broadcast Messages – Messages are sent to all reasoners at once.

13


Input Analysis and Natural Language Generation in 4M Multilingual Agenda Markup Language

DM

“Cable is not loose.”

“Is the power switch turned off?”

IAText > RDF

NLGRDF > Text

4M Dialogue Manager

14


Code switching with a multilingual grammar

4m5.defbilingual grammar

4m5en.def

English grammar4m5fi.def

Finnish grammar

4m5en_lex.defEnglish terms from ontology

4m5fi_lex.defFinnish terms from ontology

fi4.hFinnish

morphology

15


cparse parser

cparse generator

+

multilingualgrammars

+

Jena/JavaAPI

+

<rdf:RDF> <dm:Agenda> <dm:next> <dm:Greet/> <dm:next> </dm:Agenda></rdf:RDF>

Hello

Commutative graph rewriting

16


Concept Analyser

User said:“Vertti cannot access his calendar.”

This conversation is about:1. Calendars2. Mobile phones3. Humans4. Poodles5. Computers

Connects dialogue references with the things they refer to:

Keeps track of what the current conversation is about:

Mr. Vertti Hiiri

Vertti'scalendar program

Ontology

17


Reasoners

dialogue-manager

baseontology

modelbased

reasoner

diagnosticsplanning

common query-

language

extendable set of

reasoners

extendable data

schema

industry company factory/office project

common protocol

planningconceptanalysis

life-cycle supporting

tasks

case based

reasoner

extendedontology

instancedatabase

casebase

ontology-reasoner

shared database platform

Reasoning methods and techiques

18


Ontology based diagnostics

Loads in the conceptual ontology and the instance base

describing the system being diagnosed. Used domain independent search methods for localizing

the faults in the system. Uses the incremental set of observations reported by

users of the functionality of the system.

21


Stepwise fault recognition

Observation O1: Mary can't access team calendar.

O2: Jack can't access project directory.

Can Jack access team calendar?

O3: Yes Fault candidates (O1 and

O2 but not O3 )

24


Information Retrieval

... User: Internet on rikkiSystem: Toimiiko hiiri?User: En pääse nettiinSystem: Pääseekö hiiri nettiin?User: ......

Identify partslike subject, verb and

object in dialogue turns.Identify known concepts: solve

ambiguities and differencesin terminology, cross the

language barrier

...U: m4:Internet, m4:BrokenS: m4:MouseU: m4:Access, m4:NetS: m4:Mouse, ......

List known concepts and theirrelations to each other and toterms in each language

Build a query out of theconcepts in the dialogue.Retrieve best matching

documents

Sometimes you may encounterproblems with Internet access, or with input devices like keyboards and mice – or with just about anything. By simply following these trouble-shooting instructions,

Mark occurrences of known concepts with

BRIEFS ontology matching

Sometimes you may encounter <cid = "m4:Problem"> problems</> with <c id ="m4:Internet"> Internet </> <cid = "m4:Access"> access </>,or with <c id = "m4:Input">

Unannotated dialogue

Unannotated documents Document analysis

Ontology

Input analysis Annotated dialogue

Annotated documents

Information retrieval

25


Case Based Reasoning Tool

Input: a problem description as a set of RDF triplets

Target state

Observations

Case base: example problems with their solutions as xml documents

The problem is matched against the cases in the case base

The cases with similarity measure over a similarity threshold are retrieved, and their solutions are returned in similarity order

If none of the library cases is similar enough compared to the current case, the solutions of the most similar cases are used in creating further questions to the user

The similarity calculation between two cases uses ontology to determine semantic similarity

26


Human Assistance Tool

The Human Assistance Tool finds experts, administrators, and contact information according to ontology and instance base

Explicit user queries and queries initiated by Dialogue Manager: "Who is the expert of Windows", "Who is the administrator of server S1"

Passive queries representing the cumulative topic of conversation (the concepts found in the discussion) used for team building

locate all the persons in the instance base with some kind of knowledge in the conversation topic, both experts and administrators are considered

search for the people who are closest matches to the minimum spanning tree representing the topic

29


4M: Annotating documents to improve IR

Briefs

RoolToolOntology

Document

Annotateddocument

CPSL

rules

mobile multilingual maintenance man

Documents

parse tree

reportthe new term

parse graph

natural language parsing

ontological entities

rewriting system parses

client concepts

hierarchical owl ontology