![Page 1: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/1.jpg)
CUIML:A language for generating multimodal human-computer interfaceChristian Sandor
Thomas Reicher
報告人try
![Page 2: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/2.jpg)
Presentation Agenda DWARF簡介 Introduction UIML簡介 Requirement Analysis Related Work Design Example Future Work Value to our research
![Page 3: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/3.jpg)
Reference [1]Christian Sandor,Thomas Reicher(2001),“CUIML: A lang
uage for generating multimodal human-computer interfaces”, Proceedings of the European UIML.
[2]Marc Abrams,et al.,”UIML:An Appliance-Independent XML User Interface Language”,Garvin Innovation Center,VA.
[3]Micah Dubinko(2001),”What Are XForms” ,http://www.xml.com/pub/a/2001/09/05/xforms.html.
[4]Just van den Broecke(2000),”Pushlets: Send events from servlets to DHTML client browsers”,http://www.javaworld.com/javaworld/jw-03-2000/jw-03-pushlet_p.html.
[5]Martin Bauer, Bernd Bruegge, Gudrun Klinker, Asa MacWilliams, Thomas Reicher, Stefan Riß, Christian Sandor, Martin Wagner (2001),”Design of a Component-Based Augmented Reality Framework”, ISAR 2001
![Page 4: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/4.jpg)
DWARF簡介
![Page 5: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/5.jpg)
DWARF: A Wearable Computer
![Page 6: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/6.jpg)
Augmented Reality
虛擬物件與現實世界的結合
![Page 7: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/7.jpg)
DWARF 架構圖
CUIML
![Page 8: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/8.jpg)
Introduction CUIML:Cooperative User Interfaces Markup
Language. CUIML is part of DWARF project The DWARF is to develop a framework for
applications on wearable computers which consists numerous devices.
The interaction with the user should be multimodal in this situation.
![Page 9: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/9.jpg)
Multimodal System Multimodal System supports communication with
the user through different modalities such as voice,gesture and typing
Modal means “Modality” and “Mode” Modality
Communication channel used to convey information.
Mode State that determines the way information is interpreted to
extract meaning.
![Page 10: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/10.jpg)
Multimodal System(2) In the multimodal HCI,system should support user in
put and system output by various ways. Examples
Input:speech,gesture,keyboard Output:voice(VoiceXML),3D graphics(VRML),2D graphi
cs,Text(HTML).
In UIML,we want to describe HCI using an abstract language,so this language should allow us to define the behaviors of various input / output components.
![Page 11: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/11.jpg)
UIML 簡介 Create natural separation of UI design from program
ming Job. Allow many to one relationship between the two.
Transform to native markup language by XSL A measure of the power of UIML is that it can replac
e hand-coding of Java AWT/SWING user interfaces. (和 SALT有何不同 ?)
![Page 12: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/12.jpg)
Where UIML Fits InApplication database
Content (XML, audio,…)
Device Adaption
UIML
WML VoiceXML XHTML cHTML
![Page 13: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/13.jpg)
UIML Perspective
Machine language
Assembly language
“High-level” language (C++, Java)
Scripting languages
Device-dependent markup
Device-independent markup
![Page 14: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/14.jpg)
Features of UIML UIML can be rendered to many device-dependent la
nguage like VoiceXML,HTML or WML. Web-based Interfaces generated with UIML can send events but
can not receive events.(每次改變時,都要重新送一次文件 )
UIML only generate views that do not interact with each other.
![Page 15: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/15.jpg)
CUIML Overview CUIML use UIML as a starting point. CUIML is a unified user interface description
language to describe multimodal user interface in an abstract way.
CUIML suggests a MVC(Model-View-Controller) framework to solve the synchronization problems.
![Page 16: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/16.jpg)
CUIML Overview(2) By XSL transformations,these descriptions are
converted into various markup language for various devices.
The controller’s internal structure is a DFA(deterministic finite automaton),it map CUIML to various input and output device
![Page 17: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/17.jpg)
Requirements analysis Modeling information at a high level of abstraction Mapping generic information to device-dependent
presentations Synchronization of the different channels Controller configuration
![Page 18: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/18.jpg)
Related Work UIML Petri Net S.Riss Work Flow Engine XForms Mozquito
![Page 19: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/19.jpg)
Design of CUIML
![Page 20: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/20.jpg)
Runtime behavior of the components The client request a HCI by sending a HTTP request. Server renders the components and send back to
client. The rendering is done by applying XSL
transformations.
![Page 21: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/21.jpg)
Initializing the Client
Controller Config /
Information for Manipulator
![Page 22: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/22.jpg)
Generation of HCI components
![Page 23: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/23.jpg)
The generated Components(1) Controller
Synchronize the view components Keep track of current state of HCI Use WFE implementations in prototype
![Page 24: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/24.jpg)
The generated Components(2) View
Display markup languages Send event to controller The concept of describing and rendering views is adopted
form UIML.
![Page 25: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/25.jpg)
The generated Components(3) Manipulator
An enhancement over UIML. Changes to the view is done by manipulator instead of ren
dering new view. View is accessed by manipulator by the Document Object
Model(DOM).(這代表 View不能太大 )
![Page 26: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/26.jpg)
The generated Components
![Page 27: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/27.jpg)
Structure of CUIML
![Page 28: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/28.jpg)
Create Multi User Interface with CUIML
![Page 29: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/29.jpg)
Example:StickySelected
Edit this sticky
Edit here紅色代表目前被選擇的利貼
![Page 30: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/30.jpg)
DFA of the Controller
![Page 31: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/31.jpg)
Define the interface
![Page 32: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/32.jpg)
Define the events
![Page 33: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/33.jpg)
![Page 34: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/34.jpg)
Future Work To develop a CUIML IDE Security concerns of the CUIML component sent to
client side Internal structures of Controller using DFA is not
sufficient for more complex situations.One possibility will be Petri Nets or even Colored Petri Nets.
![Page 35: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/35.jpg)
The value to our research The idea of synchronize VRML 、 VoiceXML and
HTML with MVC Model.(Instead of time tag) To change views at runtime by Manipulator. The flow control language may be substitue by XLA
NG or Petri Net.
![Page 36: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/36.jpg)
Problems may solved using CUIML Voice /Animation synchronization. Protocol integration. ASR/TTS integration and its performance issues. Virtual user autonomy. The “Voice propagation range” issues.
![Page 37: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/37.jpg)
System Design Prototype(Before)
![Page 38: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/38.jpg)
System Design Prototype(After)
![Page 39: CUIML:A language for generating multimodal human-computer interface Christian Sandor Thomas Reicher 報告人 try](https://reader036.vdocuments.net/reader036/viewer/2022062308/56649ec95503460f94bd7754/html5/thumbnails/39.jpg)
Q & A