the entertainment industry meets simulation: the challenge of the holodeck bill swartout institute...
TRANSCRIPT
![Page 1: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/1.jpg)
The Entertainment Industry Meets Simulation:
The Challenge of the Holodeck
Bill Swartout
Institute for Creative TechnologiesUniversity of Southern California
![Page 2: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/2.jpg)
The ICT is intended to serve as a nexus between the simulation
and entertainment communities
![Page 3: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/3.jpg)
BizTechShow
Institute For Creative Technologies
![Page 4: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/4.jpg)
How can we best simulate reality?
• Two approaches– Military simulation– Entertainment industry
![Page 5: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/5.jpg)
Simulating Reality:Two Approaches
Military Simulation• Literal: try to include as much reality in
simulation as possible• Homogeneous Entertainment Industry• Goal-directed: create effect of real experience
by selectively focusing on elements (real and unreal)
• Hybrid Approach: combine disparate elements
![Page 6: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/6.jpg)
Movie clip
![Page 7: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/7.jpg)
To the Holodeck and beyond…
![Page 8: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/8.jpg)
Mission Rehearsal Exercise Project: New Operations
![Page 9: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/9.jpg)
Mission Rehearsal Exercise Project: ICT VR Theater
![Page 10: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/10.jpg)
What we don’t want…
Movie clip
![Page 11: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/11.jpg)
New Research Areas• Speech Recognition• Natural Language Understanding• Dialogue Management• Natural Language Generation• Speech Synthesis• Gesture Generation• Perception• Personality Variation• Sound Spatialization• Interactive Story Structure
![Page 12: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/12.jpg)
Movie clip
![Page 13: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/13.jpg)
Integration Raises New Challenges:
Natural LanguageProject Leaders: Ed Hovy (ISI), Kevin Knight (ISI),
David Traum (ISI), Shri Narayanan (IMSC)• Speech recognition: Noisy
environment• Multimodal: Face To Face
(speech+gesture), Radio• Multiple Interactors (Sgt, medic,
squads,…)– Messages tailored for multiple
addressees/overhearers• Emotion
![Page 14: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/14.jpg)
• Most speech synthesis sounds unnatural• Even the best have limited
expression (AT&T NextGen)– No command voice
• Use hybrid approach:– Multiple synthesizers with
different “voices”– Command voice
Text toSpeech
Synthesis“Squad Leaders listen up!”
Challenge: Speech SynthesisProject Leaders: Lewis Johnson (ISI), Shri Narayanan
(IMSC)
• Use pre-recorded library for highly emotional speech
![Page 15: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/15.jpg)
Challenge: Modeling EmotionsProject Leaders: Jon Gratch (ICT), Stacy Marsella
(ISI)
![Page 16: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/16.jpg)
Challenge: Modeling EmotionsProject Leaders: Jon Gratch (ICT), Stacy Marsella
(ISI)
Movie clip
![Page 17: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/17.jpg)
Challenge: Modeling EmotionsProject Leaders: Jon Gratch (ICT), Stacy Marsella
(ISI)• Problem addressed:
– Virtual humans need emotions to be believable & realistic
• Approach:– Characters have beliefs, goals, utilities– Construct plans– Events evaluated against plans– If goals thwarted (or enabled) characters
react
![Page 18: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/18.jpg)
Emotional Reaction
Movie clip
![Page 19: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/19.jpg)
Modeling Emotions: Issues
• Integrating gesture, natural language & voice to project realistic behavior
• Managing unintended interpretations
• Hollywood vs. Psychology• Movie emotions quite different from natural
emotions
• Recognizing human emotions
![Page 20: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/20.jpg)
Challenge:Story is Critical
• More than an event list• Successful stories:
– Juxtapose contrasting elements– Surprise (plot twists)– Dilemmas– Things go wrong– Build to climax (and ebb)
![Page 21: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/21.jpg)
Structuring Stories for Intelligent Agents: Issues
• Traditional story telling tricks require linear structure
• Immersed participant needs freewill
• Need “bumpers”– Possibility: director agent controls
environment & characters to limit choice
![Page 22: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/22.jpg)
Reasons for Optimism
• Hybrid approach– Use all the tools in the toolbox
• Entertainment industry as source of knowledge and techniques
• Story context constrains possibilities• Synergy from graphics, sound &
emotional engagement– The mind is a willing collaborator
![Page 23: The Entertainment Industry Meets Simulation: The Challenge of the Holodeck Bill Swartout Institute for Creative Technologies University of Southern California](https://reader035.vdocuments.net/reader035/viewer/2022070406/56649de35503460f94ad9c4c/html5/thumbnails/23.jpg)