developing a virtual piano playing environment
DESCRIPTION
Developing a Virtual Piano Playing Environment. By combining distributed functionality among independent Agents. A.Broersen. Overview. Introduction to the project Backgrounds of the concepts Design of a multi-agent platform Playing with the system Conclusions and remarks - PowerPoint PPT PresentationTRANSCRIPT
10 september 2002
A.Broersen
Developing a VirtualPiano Playing Environment
By combining distributed functionality among independent
Agents
10 september 2002
Overview
Introduction to the project Backgrounds of the concepts Design of a multi-agent platform Playing with the system Conclusions and remarks Work in progress towards the aid for
a real piano teacher
10 september 2002
Introduction: Parlevink
(http://parlevink.cs.utwente.nl)
A group on Language, Knowledge and Interaction doing research on:
Human-Computer Interaction Dialogue systems Autonomous agents Virtual worlds
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Introduction: Field
Exploring possibilities usingVR and Human-Computer Interactionin distributed learning environments.
Case: Piano Playing interactions using a real keyboard, VR,
dialogues and dataglove playing with more people at the same time creating musical tools and a teacher in a
dynamic learning environment
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Introduction: ADRI
(Artificial Didactic Recital Instructor)
GoalBuilding a framework for a piano teacher in a virtual piano playing environment
With a real keyboard connected to a piano in a 3D-world with tools for music- and piano-education in a distributed architecture
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Introduction: Requirements
ADRI should contain: different independent processes server functionality Java3D Midi for sound using Windows and Unix general knowledge/lesson structure
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Design: Agents
For flexibility different ‘Agents’ are used
What are Agents in ADRI? Different functions
different skilled agents When can you call the processes
‘Agents’? When are agents ‘Intelligent’?
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Design: Multi-Agent Platform
Connecting the agents with each other in a platform:
Simple communication (messages) between agents
On each machine one agent to control the ADRI-agents
One agent is manually controlled to handle the available agents and their different connections
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Design: Communication
Exchanging XML-messages: Communication messages
using a simple protocol
Midi messagesas quickly as possible
ADRI messagesexchange of knowledge (graphs)
using multicast functionality
‘Is someone listening?’
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Playing: The Agents
Midiatorplaying and capturing Midi sounds
Visualizeruser dialogue in a 3D World with a
visualization of a piano and notes
Supervisor(remotely) control the system
Communicatorhandle agent functionality
on one Java Virtual Machine
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Playing: More Agents
Expertprovide songs and chord recognition
Analyseranalyses the notes in a song
Observertracks the users actions and progress
Instructorgives assignments to a learner
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Playing: User Interface
In the Supervisor: Dynamic visualization and control of
the complete agent network in a graph An agent dialogue frame with the
specific options in the Agent Sending messages to agents
Different combinations of agents give different environments
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Playing: The Visualizer
10 september 2002
Conclusions: Agents
In the resulting prototype: the agents are kept ‘simple’ the graphical Supervisor provides a
good control and overview of the system
the message-parsing is fast enough for real-time exchange of Midi
there is no good structure (yet) for knowledge and lesson structure
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Conclusions: Communication
Using no standard package: The communication layer is still under
development according to different needs in the application
Quick exchange of sound and graphical events using Midi
Midi capture is a problemusing only Java
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Work in progress: Building...
Creating more demo Agents: Navigator Translator Corrector
Using advise and remarks of human experts and users
Developing plans for lessons:Using graphs for state-transition diagrams
Overview
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Work in progress: Where?
For (more) information or questions:
Parlevink, Dept. of Computer Science University of Twente, the Netherlands http://parlevink.cs.utwente.nl [email protected]
Introduction
Design
Playing
Conclusions
Work in progress
Demonstration
10 september 2002
Demonstration: Playing
10 september 2002
Demonstration: Dataglove
10 september 2002
Playing: The Supervisor
10 september 2002
Demonstration: Screen-shots
Host address
ADRI Agent
Other Agent
Route to Supervisor
Remote Host
10 september 2002
Demonstration: Screen-shots
Communication Group
Terminator
Multicast Group
Multicast Member
10 september 2002
Playing: The Midiator