multimodal research in linköping
DESCRIPTION
Multimodal research in Linköping. Arne Jönsson NLPLAB/HCS/IDA. NLPLAB. Upper row: Pernilla , Sonia, Lena, Annika, Micke, Magnus Bottom row: Håkan , Lars A, Robert, Mustapha, Arne , Lise, Anders, Lars D Missing: Nils , Bertil, Pontus, Gen. Current research. - PowerPoint PPT PresentationTRANSCRIPT
Arne Jönsson 1
Multimodal research in Linköping
Arne JönssonNLPLAB/HCS/IDA
Arne Jönsson 2
NLPLAB
Upper row:Pernilla,Sonia,Lena,Annika,Micke,Magnus
Bottom row:Håkan,Lars A,Robert,Mustapha,Arne,Lise,Anders,Lars D
Missing:Nils, Bertil, Pontus, Gen
Arne Jönsson 3
Current research
• (Multimodal) information access- Natural language interaction- Text and document analysis
• Usability of multimodal interfaces• Open-Source and development methods
Arne Jönsson 4
Dialogue system
Dialoguemanager
Generator
Interpreter
StructuredInformation
Dialogue system
Arne Jönsson 5
Ex TV programme guide
• Search for TV programs• Mediaterminal• Structured database • Spoken input• Developed from framework and method
for iterative development• With Nokia Home Communication
Arne Jönsson 6
Demo
Arne Jönsson 11
Text and document analysis
Language analysis
StructuredInformation
Unstructured documents
Q&A
MT
Arne Jönsson 12
Ex. AskRSV
• Information extraction from text document
• Simple Q&A-system• With Ida Infront
Arne Jönsson 13
Ex. taxation brochure
24 pages unstructured text
Arne Jönsson 14
Domain ontology in ASK-RSV
• Taxonomy of a limited set of objects, events, properties and relations in the tax domain
• Only IS-A and synonymy relations
OBJECT
THING
ACTOREVENT
PAPER DATE TAXATION TAX EVENT ORG.PEOPLE
TAX-RETURN ASSET DEBT INCOME TRANSACTIONCOMP. PUBL
Arne Jönsson 18
Combined system
StructuredInformation
Unstructured documents
Language analysis
Dialoguesystem
Lexicon Grammar Ontology
Arne Jönsson 19
Shared knowledge sources
• Ontology, lexicon and grammar- Different demands from IE and DS
• Design• Development
- Corpus analysis, user - Background system, database
• Automatic update
Arne Jönsson 20
BirdQuest
• Bird encyclopaedia- Tagged, templates
• Corpus with user questions- Analysed
• Based on dialogue systems framework
Arne Jönsson 26
Ötraf
• Time tables from local bus company• Multimodal system
- Dialogue- Handling of spatial and temporal
expressions
Arne Jönsson 28
Studies of multimodal interaction
• Wizard of Oz-simulation • Feedback study
- User’s experience- Interviews, questionnaires- Various dimensions: control, cooperation, ease of use,
efficiency, anxiety, hability etc- Three conditions
- No spoken feedback- Limited spoken feedback- Full spoken feedback
- 30 subjects
Arne Jönsson 29
Results
• All or nothing• More like a human in the full feedback
condition- Before the study subjects wanted to have a
timetable for the task, after they preferred a human• No difference in experienced control and
cooperation• The speech synthesis better in full feedback
mode- Same voice
Arne Jönsson 30
Open source
• Share research results between research groups- Stimulate exchange and co-operation
• Language technology readily available for industry- Increased use and insights- The companies do their own prototype
development• Robust code• nlpFarm
http://sourceforge.net/projects/nlpfarm/
Arne Jönsson 31
http://sourceforge.net/projects/nlpfarm/